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Foreword 



Two published volumes report on the 1972-1973 Panel Review 
of Products (PROP), Onei of these docuinents reports fully upon the 
procedures used and .dctloris taken by the Panel and the supporting 
staff that brought about the reported results. This report is 
issued as PR- 7 3-11 undc.r the title Selection of Exemplary Educational 
Products, by W. W. Walton, B. F. Esser, M. G. Epstein, E. H. Margosches, 
and W. B. Schrader. The second collection of information,, contained 
between these covers, reports on the 23 products selected in PROP 
1972-1973 and on the dissemination recommendations made in their 
Support by the PROP Panel and presents product descriptions as 
detailed below. 

The bulk of the present volume is taken up by nontechnical 
descriptions of the products selected for dissemination-related 
actions in the 1972-1973 review^ Attention is given to what each 
product is, what it does, and whom it is designed to serve. Special 
emphasis is devoted to evaluation evidence on effectiveness, to 
information on demonstrated benefits, to both dollar and nondollar 
costs and economies , and to the developer's dissemination efforts. 
Also included are summaries on considerations related to installation 
and use. 

Fourteen professional staff members of ETS's Test Development 
Division, in most cases those that had earlier completed analytical 
product analyses and precis on the same products, prepared the product 
.descriptions for the present report. Barbara F. Esser and Elizabeth H. 
Margosches collaborated with me in presenting the Panel's dissemination 
^recommendations and in co^ipiling the arrays of information which follow. 
A listing of the 23 products covered by this report begins on the next 
page. 



Wesley W. Walton 
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PROP 1972-1973 
PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 

DISSEMINATION RECOMMENDATIONS 

The 23 dissemination recommendations graphically displayed on the following 
pages are in essence the outcome of the 1972-1973 Panel Review of Products. The 
votes of panelists are shown in the boxes, the ten divisions of each box represent- 
ing one vote each by each of the ten Panel members. Where fewer than ten of the 
Panel members voted, either because of absence or abstention, not all divisions are 
fillad. Symbols used have been given the following meanings: 





Traditional Dissemination Action 


BiiniiiiifliiiiHiiiiiii 


Generally Favorable Vote 


imiinmifiiniuminrE 


Favorable with Reservation 

Modified Dissemination Action 

Independent Evaluation or Independent 




Review of Available Data 




Field Trials 




More Development 




Other Action as Specified 



Where more than one of the symbols above was applied to a given product, the 
vote has been divided appropriately under the relevant headings. Notations on the 
graphic displays may be decoded by reference to the list shown below. 

1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available 

Data is recommended. 
T = Field Trials are recommended. 
M = More Development is recommended. 
0 = Other Action as Specified is recommended. 

4 - "Dollar Support'^ refers to dissemination dollars. 
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PROP 1973 



PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No, 



PROP-AC 



Code No. 



Definitely 
Recommended 
(Dollar Support)^ 



' ' ' ' I L_ 



CondiZionally 
Recommended 
(Dollar Support)'* 



PANEL DISCUSSION 



_j I I t. I I 1 



J i ' » > L. 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 




Action Prior 
io Dissemination 
ollar Support)^ 



Definitely Not 
Recommended 
(No Dollar Support)"* 



PANELIST COMMENTS 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E ~ Independent EvaJuation or Independent Review of Available Data, T = Field Trials, M - More Development, O = Other Action as 

Specified. 

4 - "Dollar Support" refers to Dissemination Dollars. 
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for the 
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DISSEMINATION RECOMMENDATION 



Book No. 
36 



PROP- AC 
44 



F-292-12-K02 



The Oral Language Program 

James L. Olivero, Robert T. Reeback, and Helgi Osterreich 

Principal Investigators 
Southwestern Cooperative Educational Laboratory 
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Definitely 
Recommended 
(Dollar Support)"* 



Conditionally 
Recommended 
(Dollar Support)"* 



Promising But 
Not Yet Ready 
(No Dollar Support)** 




Action Prior 
to Dissemination 
(Dollar Support)^ 



- 1 I i l_X. 



Definitely Not 
Recommended 
(Ntj Dollar Suppt)rl)^ 



PANEL DISCUSSION 

Independently obtained evidence is needed that gains of the magnitude reported are 
educationally valuable. The gains are quite substantial and the field testing is im- 
pressive. The product is recommended for dissemination with the caveat that, as in 
case of all English-as-a-second-language programs, it should be an integral component 
of a bilingual education program and not be used separately. In addition to focusing on 
the educational value of the absolute gains, the independent evaluation should concern 
itself with the cost effectivenss of the product. 



PANELIST COMMENTS 

Specialist aid may be needed. 

There should be an independent look at the tests and needs assessment, a check on gains. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is sug^^ested. 

3 - E = Independent Evaluation or independent Review of Available Data, T = Field Trials, M = More Development, O - Other Action as 

Specified. 

4 - **Dollaf Support" refers to Dissemination Dollars. 
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The Oral Language Program 

Principal Investigators: James L. Olivero, Robert T. Reeback, and 

Helgi Osterreich 

Southwestern Cooperative Educational Laboratory 
1404 San Mateo Boulevard, S. E. 
Albuquerque, New Mexico 87108 

The Oral Language Program (OLP) is an instructional system in English as 
a second language to be used with children aged 5 to 7« It consists essentially 
of a teacher's manual and a 6-volume set of 150 lessons that cover specific 
language skills presented in a sequence determined by linguistic and language 
learning theory. Each lesson consists of a series of activities, including 
pronunciation exercises and conversation elements. Lesson objectives and a 
list of needed materials are given at the beginning of each lesson. There 
are also 5 prelessons designed to give practice in following instructions 
which are to precede the 150 content lessons, 

OLP is a one-year program and can be used successfully by 5-year-olds, 
By the time a student :s 8 years old the material seems too babyish and is 
sometimes rejected by the student. Therefore, it is recommended that the 
program be administered in kindergarten or in the first grade. That way, 
if a teacher does not complete the entire 150 lessons in one year, the 
program could be completed in a second year. 

Also included are pupil assessment devices, teacher- training procedures 
and materials, program evaluation procedures, and installation and monitoring 
procedures. Other media used in presenting the lessons include puppets, 
records, and slide-tape presentations. Other prop.^. such as classroom 
supplies, food, and toys, are used to give the lessons in^^aning. 

The program was designed primarily for use with non-English speaking 
Mexican-American and Indian pupils but has also been used with children 
who speak a nonstandard dialect of English. It is not generally recommended 
that OLP be used in its entirety with nonstandard English speakers. 

The developer has prepared materials for training teachers in the use of 
OLP and suggests the use of Quality z\ssurance Specialists who observe and 
help teachers with the program during the school year and conduct additional 
inservice training in the form of meetings after hours. These specialists 
are generally from the same school district as the teachers, 

A teacher aide may also assist the teacher with the OLP. The aide's 
function, when trained by SWCEL, is to help with the other pupils while the. 
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teacher conducts OLP lessons, to provide a cultural and language link with the 
pupils if their ethnicity differs from that of the teacher, to teach review 
lessons to those children who use them, and to teach the class when the 
teacher is absent. 

Although the OLP can stand alone, the developers think of it as part of a 
complete Communication Arts Program (CAPI) . This involves making prospective 
teachers of OLP familiar with the Laboratory's Reinforced Readiness Requisites, 
Teacher-Teacher Aide Companion Training, and Quality Assurance Program. 

The Laboratory would prefer to train all persons related to a program only 
in the initial stages. As a program progresses, the Laboratory prefers to 
serve as a consultant and provide services where necessary, always giving a 
school district an option of reducing costs through the use of on-site personnel 
and equipment. Wherever possible the Laboratory calls in "master teachers," 
experienced and specially trained classroom teachers from other districts. The 
Laboratory encourages local management of the Program, as long as quality control 
checks are maintained. The Laboratory is willing to have performance accounta- 
bility contracts with the districts which institute the OLP, although thay insist 
upon very strict adherence to the program in all aspects. The Laboratory feels 
that teachers often take liberties with the OLP, deleting lessons which in tneir 
opinion are unimportant, as well as otherwise failing to seriously acknowledge 
the English program as basic to the curriculum. Indeed, the Laboratory insists 
that unless the OLP lessons become a daily feature in the curriculum, ^ne 
students will not take the OLP seriously. They note that because zZ the re- 
quired intensity of the teacher's effort, many teachers exclude the OLP lesson 
in favor of such things as other activities in the classroom and assemblies. 

The program was field tested initially in 1968 and more extensively in 
1969-70 and 1970-71. The first field testings in 1968 of the OLP after its 
initial development made use of 24 teachers from each of 5 local school districti.^. 
in the Southwest. The pupils in the classes were Indians, Mexican- Americans, 
and Blacks. About* 1500 pupils were involved in this field trial. 

The Laboratory received feedback about implementation and use of the lessons. 
Information was obtained about such things as lessors length, pupil resprns iveness , 
use of realia, size and composition of groups, and sources of difficulty. 

Program redesign was carried out after the first field testings and during 
the 1969-70 school year the OLP was field tested again with 3 major objectives: 

1) to identify the conditions in which the program is successful, 2) to 
compare the performance of pupils using the OLP with that of pupils not using 
the program^ and 3) to determine necessary revisions in .both the program and 
the teaching strategies used. 

The findings of this field testing indicate that the mean pretest score of 
the pupils involved was below 100; i.e., 93.4, and the mean post-test score was 
137.6) for a mean gain score of 44.2. A mean gain score increase of 30 was con- 
sidered acceptable. Mean gain scores for groupings by sex, ethnic group, grade- 
level; i.e., either kindergarten or first gr^de, were comparable. The data 
further indicate that the greater the number of lessons completed, the greater 
the mean gain score. Field test findings are generally" favorable. Results 
indicate that the OLP is successful with Spanish-speaking and Indian children who 
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enter school with an inadequate knowledge of English • Teachers in general 
liked the program, the materials, and the trailing they received. There is 
evidence that children who do not use the program do not learn to speak 
English as well. 

Data for the 1970-71 field trial are based on teachers who had participated 
in summer institutes conducted by SWCEL and which covered their C^PI program. 
Usable data were obtained from 87 out of ^he 100 classes cooperating with 
SWCEL in experimental use of the CAPI , Pupil assignment to the program was 
generally done on the basis of the teachers' judgment^ but all data reported 
were based on those pupils whose pretest scores on the SWCEL Test were less 
than 130, The number of pupils involved per class ranged from 4 to 22, 

Data for the 1970-71 field trial confirm the Impression gained in 1969-70 
that the more lessons completed, the greater the mcian gain score. The data 
also indicate that OLP helped enhance pupils' self-concepts and increased 
their confidence. Teachers and the QAS both felt that teachers gained in their 
relationships with pupils and in teachin;^ techniques. In addition, pupils of 
teachers who have taught the program 2 or 3 years have a greater mean gain 
than pupils of first- time teachers. 

Pre- and post-testing is oral. All tests are administered by trained 
testers and are recorded on cassettes. The cassettes are scored in the office 
and not in the field. There is a production problem related to the testing, 
since it is carried out under conditions which are not always optimum. The 
technical problem of production is one which at the moment is difficult to 
overcome and^ because of the vast numbers of students involved, there seems to 
be no good solution to the problem of scoring. To have th^;. tests marked by 
more than one scorer would, of course, increase the expense considerably. 

The Laboratory goes to great effort to avoid claiming gains for students 
as if the gains were attributable solely to the OLP, The Laboratory acknowledges 
that students are influenced simultaneously by various programs and that 
principals provide the better teachers for the language program. Also, in most 
areas where the OLP is not in use, other programs are, so there are few adequate 
control groups. A recent project in Utah apparently was carried out with very 
tight controls. The results are said to be quite impressive in favor of the 
use of the OLP, 

One teacher with one class constitutes the minimum scope of program use. 
Teacher training as conducted by SWCEL has two parts: summer institutes and 
inservice meetings during the school year. The summer institutes are devoted 
primarily to practice in using the OLP lessons through microteaching sessions 
and by pairs of teachers. The school year inservice meetings are to help 
raise and maintain teacher morale and provide the opportunity to present, 
training components not presented during the summer. 

Prepackaged teacher- training materials have been developed -which will help 
make it possible to conduct some institutes without SWCEL staff. SWCEL staff 
does plan and/or coordinate institutes that can be taught by master teachers, 
usually teachers who have used the OLP, 

Printed materials include the teacher's manual and a 6-volun)e set of books 
containing the 150 lessons. Assorted realia in the package include handpuppets, 
records, and slide-tape presentations. Other materials that can be used in the 
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lessons include materials usually found in the classroom. Teacher training and 
materials for an entire classroom for 1 year are included in the cost charged 
by SWCEL. 

The program is complete and ready for installation. It can be purchased 
directly from the Laboratory, Cost of classroom materials for the OLP is $150 
per classroord. This is a one-time installation. The only consumables are the 
Content tests. Training in ail components of the CAPI; e,g,, OLP and RRR, at 
a 2-week institute at SWCEL comes to approximately $35O-$5O0 per person. The 
price changes in terms of the number of trainees presented and the services 
required. It is estimated that training in OL? at a one-week i.nstitute would 
be about 1/2 of the price quoted. 

The product may be used without disruption of school space or scheduling 
as the classroom teacher uses the program within the context of a self-contained 
classroom. The product may be used with groups of up to 10 pupils, on either a 
class, school, or district-wide basis. 

No specific information is available as to how many teachers and classrooms 
are currently using the OLP, Some teachers have been using the program since it 
was first developed and have been trained as "Master" teachers; they can present 
the program and train other teachers. In addition, universities such as New 
Mexico State University instituted preservice training in OLP with limited 
help of SWCEL personnel, 

SWCEL also has brochures available for distribution and personnel are 
available to conduct summer institutes. The Laboratory is also available for 
consulting purposes. 
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Book No. PR OP - A C C 'ode No. ' 

62 59 D-292-62-X02 

Social Education Grades 1-3 

Robert Randall, Principal Investigator 

Southwest Educational Development Laboratory 
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Definitely 
Recommended 
(Dollar Support)'* 



J L J_i 1 I I I— L 



' 'onditiorially 
Recommended 
(Dollar Support)"* 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 



Action Prior 
to Disseniination 
(Dollar Support)"* 



■Ji I J--1- 



l)e finitely Not 
He con) mended 
fNo HoiJar Suppnrf)'^ 



PANEL DISCUSSION 



The test data are weak and the results are not particularly exciting; nonetheless, the 
product appears to serve a need in an area in which not much else is available and 
minimal training time is required. 

Dissemination funds would be well allocated for independent evaluation efforts. 



PANELIST COMMENTS 

Teachers like the grades 1 and 2 materials. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1 974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = I-ield Trials, M = More Development, 0 = Other Action as 

Specified. 

4 - **Dollar Support** refers to Dissemination Dollars, 
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Code No. 
D-292-62-X02 



Social Education Program 



Principal Investigator: Robert Randall 



Southwest Educational Development Laboratory 
800 Brazos Street 
Austin, Texas 78701 



The Social Education Progr.5m is designed to teach social concepts to 
English-speaking children from linguistically and culturally different back- 
grounds in first, second, and third grades through 30-minute daily lessons. 
Films trips, audiotapes, puzzles, teaching pictures, and card games enable 
students who are nonreaders to use the materials. 

The materials have been designed to emphasize the positive aspects of 
the background a pupil brings to the learning situation. The focus with the 
Social Education Program has been placed upon the following objectives: to 
recognize and make provisions for the social concepts and skill deprivation 
characteristic of children from economically disadvantaged homes; to reflect 
the social experiences and value orientations of children from culturally 
different backgrounds; to provide opportunity for children to develop rational 
understanding of cultural diversity; to assure, through structure, the 
sequential development of social concepts and skills; to concentrate on the 
development of intellectual processes* 

The First Year Program focuses on the family. Sixteen units introduce 
concepts such as dependence, interdependence, roles, communication, and 
education. The Second Year Program focuses on the needs of families in a 
community. Eighteen instructional units cover overlapping social, public, 
and economic activities and introduce to the child adults involved in providing 
for community needs. The Third Yt?ar Program expands on concepts gained from 
the second year's study of a community by presenting a study of many communities, 
contrasting and comparing peoples and their cultures.^ The 13 units contrast 
various geographic conditions and represent the simple and the complex, the 
urban and the rural, and the old and the new. At each of the 3 levels there 
are auxiliary materials provided by the developer that assist the- teacher at 
the beginning and the end of the course and throughout the school year. 

In 1970-71 grades 1 and 2 materials were pilot tested. There were 1,734 
students and 64 teachers involved in the evaluation. Third-grade materials 
were pilot tested in 1971-72 with 732 youngsters and approximately 25 teachers. 
Field test sites were in Dallas, Austin, Galveston, Houston, North Forest, 
Addine, Corpus Christi , Del Valle, and Wichita Falls, all in Texas; and in 
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Lansing, Michigan; Hanna, Wyoming^ New Iberia, Louisiana; Hartford, Connecticut; 
and Clovis, New Mexico. Data collected at these sites included pre- and post- 
tests and user difficulty infcmatioru Students in May. of 1971 were administered 
two types of instruments: criterion-referenced tests developed by the Laboratory 
and the Social Studies section of the Tes' s of Basic Experiences (TOBE) , pub- 
lished by CTB/McGrav7-Hill. 

Criterion-referenced tests were developed irom specific program objectives. 
Relevant concepts were test ad by means of a content-related item. As there are 
neither national nor local norms for criterion-referenced measures, the Laboratory 
used a level of acceptable response as the criterion. Pupils participating in 
the program were expected to get at least 75% of the test items correct, and 
results are reported in terms of this criterion. In both the first- and second- 
grade groups, the youngsters met the criterion in about 80% of the cases. 

TOBE results at the first grade were slightly above the norm. (The mean 
percentilr?. was 52 for five sites, N about 550, S.D,, 25.) No second grade 
norms were available, but using first-grade norms, mean percentiles ranged at 
four sites from 62 to 72. TOBE percentile scores for Mexican-American students, 
at the end of the first grade, were lower than for black students, who in turn, 
scored lower than Anglo students. The lower percentile for the Mexican- 
Americans is not surprising in view of the fact that the Mexican-American child 
enters school with different language skills than those of English-speaking 
children. The differences are considerably less at grade twOc, with Anglos still 
scoring above the other two. No important differences were observed among ethxtic 
groups on the criterion- referenced tests. Thus the differences on the TOBE were 
accounted for by the developer as cultural experience differences. 

As stated, in 1971-72 grades 1 and 2 materials were field tested and grade 
3 materials have undergone some preliminary evaluations. The evaluation design 
for grades 1 and 2 included pre- and post-criterion-referenced and normative 
tests with a limited number of comparison groups. The data from the tests given 
in connection with the grade 3 materials have not yet been analyzed. 

Teachers completed questionnaires on user difficulty of the program. They 
thought highly of the curriculum materials and of their effectiveness in holding 
children's interest and maintaining their involvement. 

An external review of the Social Education Program was done in August of 
1972. A draft Evaluation Report with extensive data was issued in December 1972, 

One classroom teacher for each group of approximately thirty youngsters is 
required. The revised materials may be used effectively by coordinators without 
receiving training from the Laboratory. One half-day of inservice training in 
the use of the program materials is necessary for participating teachers. 

The cost of the materials is as follows: 



Grade 1 
Grade 1 



complete set — $165.00 
staff development packet 



$35.00 



Grade 2 
Grade 2 



complete set — $155.00 
staff development packet 



$35.00 
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Grade 3 — complete set ~ $320,00 

Grade 3 — staff development packet ~ $35,00 

Grade 3 — core set — $220,00 (used with teacher packet below) 
Grade 3 — teacher packet — $105-00 

Costs would run approximately $6 to $7 per pupil per annum. The materials 
are available on a noncommercial basis from Southwest Educational Develop- 
ment Laboratory'. 

The developer has indicated that dissemination funds would be used to 
identify concentrations of disadvantaged students, to prepare an inexpensive 
brochure on the Social Education Program for distribution to school directors 
in these areas, and to send Laboratory resource people out to target metro- 
politan areas to meet with potential users. 
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PANEL DISCUSSION 

An effort should be made to recommend to the developer an evaluation procedure which 
would encompass limited objectives other than global attitudinal change. For example, 
it would be useful to know what use teachers trained in the workshops make of the product 
in their classrooms. 



PANELIST C(^MMI:nTS 

Validity/evaluation data weak but difficult to assemble. 
The goals are testable. 

Doesn't really fit our currently required validation standards. 
Cognitive gains? 

Data does not mention that students learned of contributions from non-white groups. 

The real payoff would come from the development of courses on Latin American, Mexican- 
American, Jewish, and other ethnically oriented music. The product's real value would be in 
existing social studies and artistic curricula. 

NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = I ield Trials, M - More Development, O = Other Action as 

Specified. 

4 - **Donar Support" refers to Dissemination Dollars. 
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Development of Materials for a One Year 
Course in African Music for the General 
Undergraduate Student 



Principal Investigator: Vada E. Butcher 



College of Fine Arts 
Howard University 
Washington, D. C. 



The materials available consist of two course outlines, Introduction to 
African Music and Introduction to Afro-American Music , that together represent 
a one-year course; a series of monographs, units, and papers on areas and 
composers of African music; slides and tapes of rhythmic patterns and African 
instruments; bibliographies and discographies ; and tapes of works of Afro- 
American composers. The materials were created with the assistance of 
recognized African and Afro-American musicians. Resource materials are being 
addeH continuously and a collection of African instruments is available on 
loan to schools and colleges. 

The course materials developed by the Project in African Music (now the 
Center for Ethnic Music) at Howard University were originally intended for 
general education curricula in colleges, but the materials were specifically 
examined for suitability at the high school level and they have been structured 
and organized in such a way as to permit their use in secondary schools with 
minimum modification. 

The Project was originally concerned primarily with African music; the 
course in Af ro-Amarican music was added in response to a rising demand for 
Black Studies programs. The Center hopes, in addition to providing an 
opportunity to study music of intrinsic merit, to contribute to the realization 
of the general education goal of responsible citizenship by affording students 
increased understanding of and respect for their fellow American citizens and 
to correct in some measure a long-standing omission in American education by 
recognizing the contributions of nonwhite minorities to the strength and 
cultural richness of the United States. 

The materials and workshops through which teachers were familiarized with 
the courses have been tested for effectiveness in a program designed by 
specialists in educational research. 
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A Likert-type questionnaire was sent to those who attended workshops and 
a similar two-part questionnaire was sent to recipients of Project materials. 
Details are reported in the HEW final report of the Project, Development of 
Materials for a One Year Course in African Music for the General Undergraduate 
Student , September 1970. (This report also includes data on the Af ro--American 
Music course.) The more than 150 questionnaire responses indicated an over- 
whelming positive attitude toward the course, the instructors, and the 
individual components of the materials, but the Likert-type items were not 
worded in such a way as to control for the response set of acquiescence. 
Positive results would probably have been obtained even if this had been done, 
however, since the percentages of responses at the extreme positive end of the 
Likert scale were very large. 

No formal attempt has been made to validate the materials against stated 
goals' by means of rigorously designed, controlled experiments; the goals lie 
chiefly in the areas of attitudinal change and appreciation, both extremely 
difficult to measure given the current state of the art of measurement. 

A professionally certified music teacher is preferred to present the 
materials, but other teachers might be able to present the materials if they 
attended one or more special workshops. 

No product-oriented training is thought to be necessary, although attendance 
at a workshop would be highly desirable, x^ccess to native performers of African 
and Afro-American music would greatly enhance the nature of the course 
presented. 

The cost of attending a 2-week workshop given at Howard University is $44 
plus transportation and living expenses. Monographs, tapes ($6 each, with 
more than a dozen available), and a synchronized slide-tape presentation ($25) 
may be borrowed . Assuming that a school already has the services of a full- 
time music teacher, installation costs would be minimal (perhaps $5 per student 
for a class of 25) and maintenance costs even less. (it is assumed that the 
school already has tape recorders and other play-back equipment.) 

The course is suitable for an entire class, either as the curriculum for 
an existing music course or as a new course. Segments of the course might 
even be incorporated into a social studies or anthropology course. The 
minimum unit for installation would seem to be the class, although there is no 
reason why an individual student could not work with the components of the 
course on an independent study basis. There does not seem to be any upper 
limit on its use — an entire school system might adopt it. Wliile the courses 
were devised in response to a felt need for music courses to be included in 
Black Studies curricula, the music involved is inherently interesting to many 
nonblack students. 

The materials developed thus far have been shared with all workshop 
participants, with numerous visitors^, and with representatives of institutions 
who requested them^with the understanding that they would be expected to assist 
in evaluation. Exhibits of the materials developed and of African instruments 
were held at the Reston, Virginia, Festival of Black Art and at the Black Music 
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and Musicians' Project at Virginia State College at Petersburg in 1969. k 
number of concerts were held in conjunction with the Project for students 
and others at Howard. An article on the Project appeared in the November 
1971 issue of the Music Educators Journal. 



The Center would like to expand its program so that it could develop 
comparable materials for other ethnic musics: Oriental, Latin-American, 
Mexican-American, and American Indian. 

Possible problems in widespread dissemination efforts would include the 
limited numbers of authentic African instruments available for loan and the 
limited number of actual performers who would be available (especially in 
some parts of the country) either to come directly to schools or to perform 
at concerts which students in the courses might attend. 

With the availability of additional dissemination funds, problems of 
this kind might be alleviated and more high school and college teachers 
could participate in the 2-week workshops designed to familiarize them 
with the course materials. 
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PANF.I. DISCUSSION 

Despite weaknejjses in validation and other reservations, instruments such as this can help 
force action to assess teacher effectiveness. The instruinent does show technical ex- 
pertise in its development, although there are competing instruments that are probably 
equally good. Since students are inevitably going to rate faculty, they should use as 
good an instrument as they can obtain. Dissemination to student groups only is recommended, 
but other competing instruments should accompany it. A covering letter should made 
reference to weaknesses inherent in this type of instrument and the limited uses to which 
it can be put. Development should center on additional teacher behaviors and effects on 
student performance. 

PANELIST COMMENTS 

How do faculty react to its use? 

Popularity of certain courses can contaminate teacher ratings. May encourage copying 
secondary characteristics of good teachers. 

Consultant help may be needed prior to adoption. 

Could cause plenty of controversy 

Competing products at the University of Illinois, Michigan State University, Syracuse, etc., 
all with norms and validation. 



NOTATION 



1 - Condition to be satisfied is indu'^r^d in Panel Discussion. 
2* Resubmission in 1974 is suggested. 

3 - H = Independent F.valuation or Independeni Review n:" Av:nl:ibic Data, T - I-'ield Trials. M '-^ More Development, 0 = Other Action as 

Specified. 

4 - '^Dollar Support*' refers to Dissemination Dollars. 
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Instruments and Procedures for Describing 
Effective Teaching Behavior 

Principal Investigator: Robert C. Wilson 

Teaching Innovation and Evaluation Service 
University of California 
Berkeley, California 94720 

The Instruments and Procedures for Describing Effective Teaching 
Behavior has the following objectives: 1) to provide systematic, flexible, 
and economical procedures for evaluating college teachers; 2) to assist 
them by providing feedback about how their teaching is perceived by students 
and colleagues; 3) to provide a broader information base for making tenure 
and promotion decisions; and 4) to assist students in their choice of 
courses and teachers - 

The instruments consist of 4 forms, any combination of which may be use^ . 
There are short (5 items) and medium-length (36 items) forms for obtaining 
teacher descriptions from students and short (5 items) and medium-length 
(27 items) forms for obtaining teacher descriptions from colleagues. The 
student-completed forms yield numerical scores on 5 scales of teacher behavior 
as perceived by students and the colleague-completed forms yield numerical 
scores on 5 different scales of teacher behavior as perceived by colleagues. 
In addition, each of the 4 forms provides space for inclusion of items that 
might be relevant to a particular teaching situation and for open-ended 
comments. Each form also has 2 items in which the teacher is rated on a 
global scale in comparison with his colleagues in his department and the 
whole institution. Finally there is a User's Manual that describes the forms; 
gives suggestions for adapting them to local situations; suggests possible uses 
for the forms; and gives directions for administering, scoring, and interpreting 
the resultant teacher profiles. A scoring program for use on a CDC-6400 
computer is available. This program analyzes the teacher description data 
and prints out a summary report for each class of each instructor. 

The forms may be adapted for a variety of uses and situations- The most 
important use of these forms for effecting improvement in college teaching, 
in the developer's view, is the establishment of departmental and/or 
institution-wide norms and the providing of each teacher with these norms 
along with his or her individual teaching profile. 

The forms were developed as a result of part of an extensive study of 
teaching done in 1967 and 1968 at the University of California at Davis. In 
May 1967 a sample of 338 students and 119 faculty members returned separate 
questionnaires in which they identified the 2 faculty members who in their 
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opinion were the best and worst teachers. Excellent agreement was reported 
batween the students and teachers on this identification and also between the 
students in this survey and in both an earlier independent survey and a 
later — May 1968 — cross validation survey of 1,015 students. 

Each student in the 1967 sample was also asked, for each of the 2 teachers 
he named as best and worst, whether each of 158 descriptors of aspects of 
teaching was characteristic of that teacher. A factor analysis was done of 91 
of the descriptors that proved to discriminate the ''best'* from the 'Wrst" 
teachers to a high degree of significance. The 5 scales (Analytic/Synthetic 
/Approach, Organization/Clarity, Instructor-Group Interaction, Instructor- 
Individual Student Interaction, and Dynamism/Enthusiasm) used in both versions 
of the forms came from this analysis. A similar fac^tor analysis of 67 colleague 
descriptors that were most discriminating resulted in the 5 additional scales 
found in the forms for colleagues (Research Activity and Recognition, Intel- 
lectual Breadth, Participation in \:he Academic Community, Relations with 
Students, and Concern for Teaching). 

Five summary descriptions of scales used with students were included in the 
May 1968 cross validation study, along with full lists of the items from the 
earlier study. Mean scores of the summary descriptions correlated very well with 
the scores obtained from the full lists. Essentially it is the summary 
descriptions that comprise the present short forms and samples from the full 
lists that comprise the medium-length forms. 

The students and faculty asked to participate in the 1967 survey were 
picked at randbm.^ The students who responded represented 4% of the student body 
and 38% of those approached. The faculty members who responded represented 21% 
of the resident teaching faculty and 54% of those approached. This salf- 
selection may have introduced bias, however, the students responding to the 
survey were reported to be reasonably representative of the student body in 
terns of sex, class level, major field of study, and grade-point average. Also 
the agreement to a very high degree of signif icanct^ on the identification of 
the best and worst teachers between the 1967 student survey and the earlier 
survey that had a 90% return provided indirect evidence, according to the 
developers, that significant bias was not introduced. 

The study also found that "best" and "worst" teachers engage in the same 
professional activities and allocate their time among academic pursuits in about 
the same ways. Finally, it was reported that, in general, student ratings of 
best teachers showed only negligible correlations with academic rank of instruc- 
tor, class level, number of courses previously taken in the same department, 
class size, required versus optional course, course in major or not, sex of 
respondent, class level of respondent, grade-point average, and expected grade 
in the course. 

The instruments for this product were developed using all departments in a 
university, not a single department, and also using ^.ross validation. According 
to the developers, such "eatures are not found in many other similar instruments, 
although they do feel Vll^I there are other good instruments of this type in 
existence. 

In 1972-73, the instruments were in use in more than 100 colleges and 
universities. By January 1973, over 1,000 requests for information had been 
received by the developers. A follow-up study of these requesters was done in 
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May and June 1972. In this study, 846 questionnaires were mailed on May 12 
to individuals who had requested information before then, of which 344 were 
returned by the cutoff date of June 5, The cutoff date was early because 
of the scheduled termination of funds for the project on June 30. Those 
returning questionnaires were representative of the total sample by position 
(47^ professors, 28% administrators, 16% research group?, and 10% other 
including student group, librarian, and committee chairmen), by type of 
department, and by type of institution (55% universities, 26% 4-year colleges, 
9% 2-year colleges, and 10% other). 

The respondents reported a variety of reasons for requesting inforraation and 
channels by which they found out ah out the study. Also, 63% planned to use 
the materials as background and 56% for possible ideas for developing their 
own instruments, and only 20% planned to use the questionnaires themselves. 
However, after seeing the questionnaires, 95% found the materials useful for 
the purposes requested; 40% recommended the materials to someone else; and 
the question "Have you actually used or do you plan to use the questionnaires 
or some portion of them?" was answered "Yes" by 26%, "Maybe" by 54%, and "No" 
by only 20%. Faculty were more inclined to use the questionnaires than ad- 
ministrators or researchers. Those answering "Yes" to the question above 
also planned that the results would be. given to individual instructors (76%), 
given to department (53%), given to administrators (34%), considered in 
tenure decisions (31%), given to students (23%), considered in teaching 
awards (11%), and other (26%). 

^ The product could be implemented by an individual teacher, a department, 
or a whole institution. No organizational changes should be requi-^ed. The 
number and type of personnel required for implementation depend on the scope. 
If implemented by a department, the administration, reduction of data, and 
interpretation can be handled by a Committee. If implementation is 
institutional, centralized data-reduction facilities may be necessary. These 
may often be provided by an existing unit such as a research or testing office. 
No specialized training for the personnel is necessary; the necessary Information 
for implementation is provided in a user's manual. For large-scale imple- 
mentation, advice by a member of the psychology, education, or sociology 
department is recommended. 

The materials required are the User's Manual and the forms. The Manual 
and sample forms are sent free of charge from the principal investigator. 
The user may then reproduce and adapt them to his own situation. 

No special facilities are necessary if data reduction is done by hand. If 
optical-scanning response sheets or punched cards are used, access to the 
appropriate machines is necessary* However, the total cost of an evaluation 
depends on many factors such as the numbers of students and teachers involved, 
whether departmental or institutional norms ore computed, and the form in 
which the results are disseminated. The developers estimate a cost range 
of about $5 to $25 per teacher evaluated. 

Since many faculty members have reservations about the institution of 
formal procedures for evaluating teaching, successful implementation requires 
early and continuous consultation with the faculty. Safeguards must ha 
provided to insure that the information gathered takes into account individual 
and local circumstances. 
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Dissemination efforts in the past have not been very vigorous. These 
efforts have included: 1) A summary report of the project that appeared in 
the monograph, The Recognition and Evaluation of College Teaching (19/0) by 
Kenneth Eble for the Project to Improve College Teaching of the AAUP; 2) a 
cut-out coupon for requesting materials that appeared in the D & R Report 
of the Center for Educational Development and Res earch (CEDaR) j 3) articles in 
newsletters and journals; 4) speeches by project staff to professional societies 
and 5) word of mouth. In ad(*.itjon, 2000 copies of the monograph Evaluat io n of 
University Teaching (by Hildebrand, Wilson, and Dienst) were distributed through 
Kenneth Eble and the AAUP. Ongoing work on the product has been terminated and 
there are no more funds a* mailable for dissemination. Informal efforts such as 
those listed above will probably continue. 
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PANEL DISCUSSION 

Although clearly not ready for extensive dissemination, this product would be particularly 
important in any focus on career education. There is some indication of positive results 
with the use of the product, although the n's were small. Further field trials might be 
done without expending dissemination funds. Otherwise, a funded independent evaluation 
seems to be the best plan of action. 



PANELIST COMMENTS 

No on-the-job data for the control group. Positi^ie and negative results are highly mixed. 

Only metal forming begins to be ready — there are no real data showing the value of 
the other clusters. 

A well-equipped vocational- technical high school is needed to use this product without 
high installation costs. 



Only 8 of 10 panelists present during final balloting. 

1 - Condition to be satisfied is indicated in Panel Discussion. 
2- Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = Field Trials, M = More Development, O * Other Action as 

Specified. 

4 - "Dollar Support'' refers to Dissemination Dollars. 




Educational Testing Service, Princeton, New Jersey, February 1973 
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The Cluster Concept Program (C^P) is form of "vocational education which 
attempts to prepare the individual for entering gainful employment in a number of 
occupations which have sufficient commonality in pHill requirements to permit 
a high degree of mobility within the vocational cluster. The program is aot 
conceived of as a means of training master craftsmen in any one occupation. 

In Phase I of program development, begun in 1965, the cluster concept was 
investigated as a form of vocatic^n.-^l education. Three clusters were defined: 
1) construction, 2) electro-mechanical installation and repair, and 3) metal- 
forming and fabrication. These operations provided the basis for such things 
as course outlines and objective achievement test items. The chief aim of 
Phase II was to identify and train competent teachers' for implementing cluster 
concept pilot studies. Eleven teachers were selected and received special 
training in the Spring semester of 19o7 a'c the University of Maryland. Phase III 
(1967-68) was concerned with the implementation and evaluation of the program. 
Phase IV, which was not funded, proposed an intensive validation study. 

Essentially, CCP focuses on the mastery of tasks needed for entry in the 
occupations within each cluster. For example, one task for welder, which is 
in the metal-forming and fabrication cluster, is "arc welding ferrous metals 
with an A.C.-welder to produce a horizontal tee joint." The performance on this 
task can be judged by looking at the human requirements in each of the following 
areas: communications, measurement, mathematics, science, skills, and information. 
The communications requirement is "reading blueprints co determine type of weld 
required." The measurement requirement is "measuring stock with a rule or scale 
to determine length." Each human requirement is assessed for its degree of 
commonality rith other occupations in the cluster. For example, the communications 
requirement in welding is common to all occupations in the cluster; the science 
requirement is common only within welding. 

The appropriateness of tasks for particular occupations was established 
through a panel of representatives of industry and busi.ness consisting of 
supervisors, owners, and union officials. This panel determined the appropri- 
ateness of the tasks and assigned them levels. Level I tasks were those required 
for entry into the job; Level II weio advanced requirements that would be needed 
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within 6 months after job entry. These levels were constantly monitored for 
current applicability. 

Pretests and post-tests were administered to 4 control and 4 experimental 
groups in each cluster for the purpose of determining the impact of CCP on 
student behaviors in the cognitive, psychomotor, and affective domains. The 
total number of students in the experimental groups was 143;. in the control group, 
150, Statistical proceduxes were applied to scores on a cluster concept achieve- 
ment test and the Mechanical Reasoning Test (part of the Differential Aptitude / 
Test battery). Data were analyzed by school. For the achievement test scores^^ 
the analysis showed that: 1) Three construction programs (schools) out of 4 
achieved significantly higher scores than the control groups on the post-tests^,/' 
Three schools showed significant gains between pre- and post-tests. 2) In all 
4 schools implementing the me tal- forming and fabrication cluster program, the 
experimental groups achieved significantly higher scores than the control 
groups on the post-tests. All schools showed significant gains between the 
pre- and post-tests. 3) Three schools implemented the electro-mechanical 
installation and repair cluster. Because of numerous failures, operations in 
one school were terminated. Neither of the remaining schools achieved significant 
gains or significantly higher post-test scores than the control group. 

Data derived from the Mechanical Reasoning Test indicated that neither CCP 
nor traditional vocational education had significant effects on the development 
of the abilities required to solve problems of applied science and technology 
as measured by the test. 

In the affective domain, the pre- and post-test scores on the Minnesota 
Vocational Interest Inventory indicated no clear patternrs or trends. However, 
the CCP groups showed more flexibility of occupational choice than did the 
control groups. 

Evaluation of the adequacy and appropriateness of course content was 
accomplished through the use of checklists and interviews by field staff, with 
the following results: 1) In the construction clustei schools, 34% to 67% of 
the tasks in the occupations within the cluster were completed. Of the tasks 
completed, 50% to 66% of the tasks needed review in anticipation of revisions 
or restructuring. 2) Fifty to 67% of the tasks were completed in the metals- 
forming and fabrication cluster. Data gathered indicated that it was necessary 
to review 25% to 34% of the tasks. 3) -In the 2 electro-mechanical cluster 
schools, approximately 50% of the tasks were completed; of these, two-thirds 
needed review. 

Support dimensions and teacher effectiveness were evaluated through 
observations and interviews. In addition, the following data-gathering devices 
were used: 1) personal vitae and records of cluster teachers; 2) visual media 
such as drawings, plans, photos, and written descriptions of practical work 
performed while implementing course outlines; 3) student progress charts; and 
4) student evaluation charts. Field staff, school administrators, and school 
personnel participated in collecting data. 

Support dimensions evaluation showed that because of the prevalence of a 
requisition-acquisition time lag (period of time between ordering materials 
and receiving them), the sequence and balance of the structured clusters were 
disturbed. Where: administrative support was strong, these problejiis were 
gradually resolved, but took their toll in leaving many of the occupational 
tasks untried. 



ERLC 



AC72 



Teacher evaluation revealed that in the construction cluster, teachers 
with degrees in industrial education tended to conduct the program in a 
superior way. Most teachers emphasized those occupational tasks for which 
the vocational classroom was originally designed and equipped. One of the 
4 teachers provided the students with all the tasks specified in the 
instructional materials. 

All the teachers in the metal-forming cluster had B.S. degress, varying 
amounts of graduate course work, and industrial experience. This group of 
cluster teachers was evaluated to be most effective in meeting the goals of 
the program. There was also a great deal of teacher interaction with 
industrial firms which resulted in free materials, technical information, 
and summer jobs for the students. Both electro-mechanical teachers were 
resourceful in obtaining materials and equipment from local industries. They 
tended to emphasize air conditioning, refrigeration, and typewriter repair, 
with little or no consideration of the other facets in their cluster. 

Special teacher selection and training are vital to the successful 
implementation of the program; however, screening and training programs are 
not now available at any university or educational center. Community and 
industrial involvement is also necessary so that clusters and occupations 
are relevant to real-life requirements and so that continuity between school 
and work is established. 

The program requires special classroom layouts and facilities and highly 
specialized equipment. A rough count of the supply list for the electro- 
mechanical cluster produced no less than 170 different equipment and supply 
items. The costs of the CCP curriculum materials are minimal, but the cost 
of equipping a laboratory for one or more of the clusters is $25 ,000-$40 , 000. 
Clearly, a well-equipped, vocational- technical high school is needed to 
use this product without high installation costs. 

The principal investigator has noted in his summary of Phase III findings 
that the most significant factor which could undermine the successful imple- 
mentation of the program is inadequate supplies, materials, and equipment. 
The requisition-acquisition time lag suggests that all cluster programs should 
be in operation for several years before evaluation is attempted. The absence 
of consideration and planning for female students may be a serious drawback 
if such a program is instituted in schools on a nationwide basis in the future. 

A teacher preparation curriculum, curriculum materials, and instructional 
plans are available from ERICv. No additional dissemination efforts are planned 
at this time. 

Although clearly not ready for extensive dissemination, this product would 
be particularly important in any focus on career education. There is some 
indication of positive results with the use of the product, although the number 
of students in the study was small. Further field trials might be done without 
expending dissemination funds. Otherwise, a funded independent evaluation 
seems to be the best plan of action. 
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PANKL DISCUSSION 

Additional evidence is needed that this product is superior to other similar products* 
Most panelists felt that the costs are very high. Consequently, further independent 
comparative evaluation is called for with regard to such things as cost effectiveness, 
and potential side effects, such as a postive influence on teachers, and gains in reading 
skills by pupils. j 



PANELIST COMMF.MS 



If suminative evaluation cannot be done until the student completes 6 years, IPI is 
inappropriate for pupils from low income families because of high mobility. 



NOTATION 



1 - ("ondilu»n to be satisfied is indicated in Paiicl Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - K = Independent (evaluation or Independent Review of Available Datj, 7 J ieid frjaLs, M = More Development. O ~ Olhci Action as 

Specified. 

4 - ''Dollar Support" refers to Disseminaijon Dollars. 
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Code No. 
D-632-49-P01 



Individually Prescribed Instruction 
Mathematics 



Principal Inves tigator : Robert Glaser 

Learning Research and Development Center 

University of Pittsburgh 

208 M. I. Building 

Pittsburgh, Pennsylvania 15213 



IPI Mathematics is a program of instruction in mathematics for grades K-6 
which is designed to individualize instruction through sequenced student 
performance objectives, related instructional materials, and diagnostic 
instruments tied closely to units of instruction. Currently, 363 instruc- 
tional objectives exist for IPI Mathematics across 10 broad content categories. 
These categories are: numeration, addition-subtraction, multiplication, 
division, fractions , money , time , measure , geometry , and applications . The 
363 objectives are divided into "Levels" which correspond roughly to grades. 
For example, there are 40 objectives in Level A, which is roughly equivalent 
to Kindergarten. 



Materials are of two types: classroom instructional and assessment 
materials and training materials for teachers and administrators. Assessment 
materials consist of placement tests, pre- and post-unit tests, and curriculum- 
embedded tests. The test questions are sometimes multiple-choice and sometimes 
open-ended. Learning materials consist of 85 IPI-constructed units covering 
the various behavorial objectives by placement level. Audio tapes correlated 
with instructional units are part of the learning system. 

Students progress at their own rate (generally without instruction) througl 
the various units. When a student achieves an acceptable score on a unit post- 
test, a new "prescription" is written. A unit is typically several pages long 
and focuses on a few objectives. The unit pretest consists of several items 
related to the objectives and a criterion score (say 85%) is set for passing. 
If a student does not reach the criterion score he is assigned the appropriate 
instructional unit. When finished with the unit, the student takes the unit 
post-test. If he achieves the mastery score set for the unit» the teacher 
gives the student a new assignment. If the student does not achieve the 
mastery score, he is recycled through the same unit. The materials in these 
instructional units are similar to programmed materials. Classrooms are 
"managed" by nonitoring student progress through the instructional units by 
use of the assessment materials. Student profiles, placement profiles, and 
prescription writing sheets are necessary to handle routine forms and administei 
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unit tests. The traditional role of the classroom teacher must be considerably 
altered to effectively administer IPI . New roles for the teacher include 
classroom management, prescription preparation, arranging for personal or peer 
tutoring, and small group instruction. IPX is usable over a wide range of 
abilities. 

In 1965 the IPI Mathematics products were installed in Oakleaf School, 
Pittsburgh, Pennsylvania, mainly to test the feasibility of individualized 
instruction in mathematics. Revisions since this initial* introduction of IPI 
have been toward the Mark II version, which was marketed in September 1972. 

The original IPI development plan called for 5 phases. Phase 1 was the 
identification of 5 school districts which would agree to demonstrate and test 
IPI. Phase 2 was the establishment of a demonstration training school in 
addition to the Oakleaf School. Phase 3 involved the training of staff and 
installation of IPI in 5 pilot schools. Phase 4 was the evaluation and necessary 
revision of the IPI program. And finally. Phase 5 was to spread the use of 
IPI to other schools. 

During Phase 1, the 5 demonstration and development schools served different 
socioeconomic populations. Schools in Pennsylvania, Delaware, and New Jersey 
participated in Phase 1 and school staff were trained at the University of 
Pittsburgh in 1966. Many of the changes in materials that were made during 
Phase 1 were suggested by teachers iu participating classrooms. During and 
following Phase 1, several sources of information were used to monitor IPI in 
the development and pilot schools. These were 1) data from placement tests 
and pre- and post-unit tests, 2) curriculum-embedded tests, 3) rate of learning 
data, as evidenced by the time required to go from point A to point B in the 
hierarchical sequence, 4) use of trained observers, 5) parent and teacher 
questionnaires, 6) student interviews, and 7) standardized tests. 

Following the 1966-67 tryout of materials, the informal worksheets were 
revised and the 1967-68 edition of IPI was published by Appleton-Century-Crof ts . 
Prior to publication, extensive changes in format and in the ordering of objectives 
were made based on the formative evaluation. Changes in IPI have been made during 
the intervening summers since 1967-68. However, most of the changes were related 
to the preparation of IPI as a commercially acceptable product rather than to 
major changes in content. 

Considerable data exist concerning the achievement: of IPI students on 
standardized tests. There are over 25 references to summative evaluations (some 
comparative) using both standardized and affective indicators. The Iowa Test of 
Basic Skills was used in a number of studies. According to summary data provided 
in Product Development Report 17 (January 1972) : 1) IPI students achieve as well 
as or better than non-IPI students on standardized measures and achieve higher 
than non-IPI students on measures designed for IPI; 2) parent reactions have been 
positive; 3) IPI is effective over a variety of populations - rural, disadvantaged,- 
regular, and retarded. IPI has been extensively monitored and tested in many 
states. A 1971 report prepared on IPI by Tom Kriewall of the Institute for 
Educational Research provides an independent judgment of IPI. The report was 
prepared for a member school district using IPI and concluded that pupils who 
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begin their study of IPI in grade 1 demonstrate by grade 3 significantly 
superior achievement compared to non-IPI pupils. All pupil groups sampled 
indicated generally positive attitudes toward mathematics, with those 
pupils in first-year IPI materials exhibiting the most positive attitude. 
IPI teachers' opinions were reported as strongly divided — the majority 
strongly supported IPI but a minority just as strongly opposed it^ Teachers 
did not feel IPI to be useful in learning ind j.vidualizing techniques. (Note: 
The developers claim no real suramative evaluation on IPI can be done since 
it is under continuous modification. A student who started in kindergarten 
in 1967 with the first published version of IPI would by 1972 be in the 5th 
grade of IPI (assuming continuous exposure). Hence, no student has yet 
been exposed to a complete IPI course, K~6.) 

In addition to the regular classroom teacher, several paraprof essional 
aides per class are necessary to do such things as keep records and administer 
tests* Teachers and administrators must be trained to use IPI and to interpret 
and utilize test results. Teacher training materials have been prepared for 
IPI staff to train local administrators to train teachers. One week is required 
for the administrators to train teachers and three days are required to train 
the teacher aides. Organization is required to provide' flexible scheduling, 
to coordinate professional and paraprof essional responsibilities, and to prepare 
the principal to cope with somewhat different problems. 

No special facilities or equipment are needed. However, the various 
instructional materials are numerous and require a considerable amount 
of space. An area is needed for storing materials, taking unit tests, 
record-keeping, and individual student study space. Several paraprof essional 
aides per class are necessary. Audio tapes require the use of several 
cassette players per classroom. 

Budget: Based on 450 pupils 



Administrative training 
Training for 18 teachers 
Teacher materials 
Aide materials 

Cost of 5 aides at $3,000/aide 



$ 300.00 
5,400.00 
180.00 
25.00 



for year 



15,000.00 



Student materials 
Shelving 



4.500.00 
500.00 



Total Budget 



$25,905.00 



Other Cost Information 



$6.50 per student for 150-400 students, publisher trains one person 
$6.50 per student for 401-800 students, publisher trains two people 
$6.50 per student for 800+ students, publisher trains three people 
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In ctddition to the instructional materials, other publications are: 

1) Aiding IPI Mathematics ; A Manual for Teacher Aides in IPI 

Mathematics • 

2) Teaching IPI Mathematics : A set of training materials for 

teachers . 

3) Audio-Visual Sources for IPI Mathematics. 

Materials are available from: 

Research for Better Schools, Inc. 
Philadelphia, Pennsylvania 

In December of 1971 IPI Mathematics was being used in over 200 school 
systems. Research for Better Schools is in charge of disseminating IPI. 

The Mark II version was released for use during 1972-73. A nationwide 
system of demonstration schools, with schools in each state equipped to 
demonstrate IPI in operation and train adminstrators and staff, is planned. 
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PANEL REVIEW OF PRODUCTS 
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DISSEMINATION RIX OMMKNDATION 
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Coilc \tr 
F-992-12-Z^01 



The Sullivan Reading Program 

M. W. Sullivan, Principal Investigator 

Sullivan Associates 

Menlo Park, California 



1 I i I t t I I I I 



Definitely 
Recommended 
(Dollar Support)'' 



B' 



LiiLLiLLi 



Conditionally 
R<;commended 
(Dollar Support)'' 



_J L_J k t_J L. 



Promising Bu( 
Not Yei Kt-ady 
(No Dollar Support)** 



Action JVior 
to DiKseminadon 
(Oolldr Support)-* 



Dctiniicly Not 
Kc CO mm ended 
(No Dollar Support)** 



PANF.L DISCUSSION 

No funds should be allocated except for dissemination into areas where a presently unmet 
need is recognized by NIE. The product ,is successful and the data on it look good. It 
seems appropriate to use subsidy help to get the product into places where special needs 
exist. The general question of dissemination backing of commercially marketed products 
needs careful policy consideration by NIE. The .American Institutes of Research report 
on the product should be part of any dissemination package. 



PANELIST COMMF.NIS 



Appears positive, especially for use in the inner*-city. 
Extensive data, well controlled verification. 



NOTATION 

3 panelists abstained from vo t ing . 

1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

.3 - E = Independent Evaluation or Independent Review of Available Data, T = 1-ield Trials. M = More Development, O = Olher Action as 
Specified. 

4 - "Dollar Support** referi to Dissemination Dollars. 
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Code No. 
F-992-12-2.01 



1 



The Sullivan Reading Program 



Principal Investigator : M. W. Sullivan 

Sullivan Associates 
Menlo Park, California 



The Sullivan Reading Program consists of several different series of 
materials with different names and different purposes. However, the major 
part of the materials is made up of two basic series, the Sullivan Reading 
Program and the Programmiid Reading Series. These are intended primarily for 
children at the kindergarten to third-grade levels but they can also be used 
in remedial reading instruction for children and adults of all ages. The 
Sullivan reading materials provide an individualized reading program that is 
programmed and is also based on what linguists have found out about how 
children learn. Tb? materials for both series are planned so that the basic 
skills needed for reading the English language are presented in logical order. 

The first of the two series listed, the Sullivan Reading Program, is 
designed primarily to help children who have reading problems. It is divided 
into 5 numbered series, each corresponding to one school year. The series for 
the lower grades consists of several programmed texts accompanied by several 
readers, while the series for the middle grades has only texts. A teacher's 
manual and a test booklet are included in each series. Also available are 
supplementary tapes for those pupils who need special help and a class record 
book for teachers to record pupils' progress efficiently. 

The Programmed Reading Series is made up of a prereading program and 3 
basal reading series. The prereading program is intended for children in 
kindergarten and first grade. It makes use of alphabet cards, sound-symbol car 
an alphabet chart, a prereader, and a teacher's guide. Series I, which is used 
for grade 1, consists of Programmed Reading books, accompanying storybooks, a 
student test booklet, and a response booklet. Series II for grade 2 aad 
Series III for grade 3 contain the same kinds of materials as Series I. This 
program is designed more for general readers than the other program, which is 
suitable for inner-city pupils and those with reading problems as well as for 
general readers. 
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All the Sullivan reading materials, with the exception of the Programmed 
Reading Series, are used in a systems approach to reading known as Project READ. 
This project provides reading materials, educational consultants, a parent 
information and involvement program, and teacher training. 

A project making use of the Programmed Reading Series, Project SPAR, is 
intended to provide a total system for achieving both accountability and 
performance in reading. It also provides inservice and preservice training for 
the total school staff, not just teachers. It involves several other major 
components in addition to the Programmeu Reading Program. 

After developmental testing in Virginia during the early stages of 
preparing the materials. Programmed Reading materials were tested in temporary 
form in Mt. View, California during 1961-1963. Several regular classes and 1 
re.Tiedial class were involved: a first-grade group of 9 girls and 13 boys, a 
second-grade group of 6 girls and 7 boys, and a remedial group of second- and 
third-graders which included 5 girls and 6 boys. A control group was comparable 
to the experimental group in terms of age, sex, IQ, and socioeconomic background 
(middle class). Both groups were tested at the beginning of the school year 
with the appropriate Lee-Clark or Gates reading test. They were retested in 
January and June. The experimental first-grade group showed an average growth 
of 2.0 years and the control group, 0.9 year. In the second-grade, the growth 
was 1.4 years compared with 0.9 year. The remedial class achieved an average 
growth of 2.3 years. Teacher and pupil comments were elicited and materials 
were revised accordingly. 

During 1965-1966 the Programmed Reading materials were used in one 
first-grade class in each of 4 schools in Colorado Springs, Colorado. A control 
group was set up in each school. The experimental group of 114 was comparable 
to the control group of 113 in terms of reading readiness and IQ. At the end 
of the year, the Metropolitan Achievement Primary I Battery was administered 
to both groups. Statistical differences in favor of the experimental group 
were found in the areas of word knowledge, word discrimination, and total reading. 

Some schools and districts performed their own evaluscions after using the 
materials or had outside auditors do so. One of the evaluations using outside 
auditors was in Dallas, with a sample size of 8,000 covering several grade levels. 
This study. Evaluation of the Targeted Achievement in the Reading Program ^ 
covered 4 aifferent programs used in the various schools of the Dallas District. 
Information about, all programs is included, but summary statements indicate that 
the largest overall gains appeared to be made by students enrolled in the BRL 
Sullivan Reading Program. This impression was reinforced by results on criterion- 
referenced tests that substantiated results on standardized tests. 

A report on Project READ evaluation, using the Sullivan Reading materials, 

and conducted in Inglewood, California, indicated that that the experimental 

groups in grades 1 to 3 attained consistently higher me£,r.s on the Cooperative 
Primary Reading Test used as the post-test than the cQUtrols; i.e^,, for Grade 1, 

the mean was 2.0 for the project (experimental) group and 1.8 foi: the controls; 

for Grade 2, the means were 2.9 and 2.6, respectively; and for Grade 3, 3.3 and 
3.0. 
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Other research reports on the Programmed Reading Series indicate that 
those students who had used the Sullivan materials generally had higher means 
on language and reading post-tests tnan the control groups with which they 
were compared. 

The reading materials may be used by a single teacher in a self-contained 
classroom. T( r- her training is recommended as very helpful but is not absolutely 
necessary since the teachers' manuals that accompany each set of mp.terlals 
prpvide complete information on using the materials successfully. Teacher 
training and consultation are provided as part of Project READ. Consultants 
will provide help before and during use and will also demonstrate the most 
effect! ye ways to use the materials. Consultants are also available through- 
out the school year to work with students and teachers. 

The programs are individualized so that pupils and/or the teacher can set 
up schedules and space to suit their own needs. No changes in staff or 
facilities are necessary and no equipment other than that supplied by the 
product itself is required. There should be no system disruption and materials 
may be used on a classroom, school, or district basis. 

The materials described previously are required to impleirient the Sullivan 
Reading Program and Programmed Reading program and all are available. Since 
some of the programmed texts require that.^the children write in the book, these 
must be considered consumable, but other materials would entail a one-time cost. 
The basic texts for the 5 series of the Sullivan Reading Program cost $1.69 
each, while the readers for s :jries 1 to 5 cost $.99 each. The teacher's manual 
costs $.99 while each^test booklet, class record book, and placement examination 
costs $.49. A manual' describing Behavioral Objectives that pertain to the 
Reading Program costs $1.49 and supplementary tapes that can be used with pupils 
needing extra help cost $18.00 for the set. 

Project READ, which uses the Sullivan Reading materials, costs $20.00 per 
student for a full year's program, or $15 per student for a one-semester or 
summer session. If a school district can place an order ior $40,000 or more 
for BRL Sullivan Reading materials, the district is eligible for Project READ. 
The cost per pupil is $.20 less for Project READ than without added services. 

Specific price informaMon for each part of the Programired Reading is not 
presently available, al^Wglr information about Project SPAR which uses the 
Programmed Reading materi'S^e indicates that the cost is $15 to $30 per pupil 
depending on what is contracted' for . 

*^ ^ 

Most materials, except' for^^he Programmed Reading program, are published 
and marketed by Behavioral Research Laboratories, Palo Alto, California. 
Programmed Reading is published and marketed by the Webster Division of 
McGraw-Hill Book Company, New York, New York. 

Approximately 5 million peonle are using Sullivan reading materials of 
various kinds, with at least 100 large urban school districts using Project 
READ, the all-encompassing systems approach to reading. Generally, it seems 
that the Sullivan Reading Program (BRL) is used by inner-city schools for 
remedial purposes, while the Programmed Reading materials (McGraw-Hill) seem to 
be used by schools with pupils less likely to have serious reading problems. 
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In regard to .dissemination, it seems that all parts of the country are 
reached ^through mailihg and advertising campaigns. Verv little direct 
contacting of schools to sell the materials is done; most sales are buyer- 
initiated. • ■' 

\ .k • • 

■; „ " ' .. "o 

• ft.. S, 'i, .'. . 

Several areas w^|e mentioned as being comparatively untouched. For the 
Programmed materials C%^aw-Hill) , these areas were the Pacific Northwest 
the Southeast, affd th^ North Central regions around Wisconsin and the Dakotas 
For the Sullivaa (ERL«^ading Materials, the untouched areas of the country 
included the Pacijic^tfihwes/ New England. Alaska was mentioned as just 
bsgmnmg to open u^^ah'd inqtiiries are being received from that state. 
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PANEL REVIEW OF PRODUCTS 
for the 

National Institute of Education 
DISSEMINATION RECOMMENDATION 



Book No. PROP- AC C(hIc \<h 

43 92 F-691-01-A01 

Home-Oriented Childhood Education Program for Rural America 
Roy Alfcrd, Principal Investigator 
Appalachia Educational Laboratory 



_LJ 1 1 > 



Definitely 
Recommended 
(Dollar Support)"* 



Black and White Materials 
and Home Visitors 

B' ( 



immi 



Conditionally 
Recommended 
(Dollar Support)^ 



111 111 



Promising But 
Not Yet Ready 
(No Dollar Support)*^ 



Development of Color 
Videotapes 



1 I 1 I l_J 



Action Prior 
to DisseminiJtion 
(Dollar Support)"* 



Definitely Not 
Recommended 
(No Dollar Support)*^ 



PANEL DISC USSK)N 

The major interest is in seeing the simple (black and white) visual materials find use 
before resources are spent on more complex color materials. The dissemination support 
recommended Is for the black and vhite materials and home visitors only, without the use 
of the mobile unit, although there is some interest in the mobile unit, especially because 
it provides for social interaction not otherwise available. A ''product'* such as this needs 
a reasonable chance to prove its effectiveness, although development funding seems beyond 
the scope of a dissemination program. A minority voted for development funds for producing 
color videotapes; there was concern that entirely new production would be needed for these. 

PANELIST COMMENTS 



True cost figures should be obtained by an independent audit. 
In how many homes is television available? 



NOTATION 



1 - Condition lo be salisfied is indicated in Panx:! Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T - Field Trials, M -■- More Development, O = Other Action as 

Specified. 

4 - '*Dollar Support" refers to Dissemination Dollars. 
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Code No. 
F-691-01-A01 



Home-Oriented Childhood Education 
Program for Rural America 

Principal Investigator: Roy W. Alford 

Appalachia Educational Laboratory 
P.O. Box 1348 

Charleston, West Virginia 25325 

The Appalachia Preschool Education Program (APEP - also referred to as 
HOPE — Home-Oriented Preschool Education) is a plan for a home-oriented 
instructional system designed for 3-, 4~, and 5-year-old children. The 
program was designed as an alternative form or type of preschool education 
experience for children living in sparsely populated rural areas and was 
intended for large-scale implementation," i . e . , school districts, regions 
or state-wide systems. 

The program itself consists of three basic elements: 1) 30 minute 
television lessons which are broadcast into participants' homes five days a 
week, 2) a weekly home visit by a paraprof essional who discusses the program 
with parents and children and who delivers materials for them to use, and 
3) group instruction given once each week in a mobile classroom. An 
important aspect of the program is the active involvement of parents in teach- 
ing or aiding their children in the learning process. A second basic considera- 
tion involves the accessibility of television to participants. 

The primary objective of the Appalachia Educational Laboratory in devising 
HOPE was to develop a new approach to preschool education adapted to the needs 
of children in rural areas as an alternative to the traditional kindergarten 
which is stationary and requires the transportation of the children to the 
school. It was to be implemented by multi-district units and was to make 
extensive use of technology, media, mobile facilities, and differentiated 
staff. The general objectives of this program are to increase the learning of 
3-, 4-, and 5-year-old children by means of televised instruction (to provide 
new experiences, to encourage children to want to learn, to initiate basic 
skill instruction, and to provide parents with a first-hand observation of the 
instruction of the children), mobile classroom instruction (to initiate social 
interaction in small groups, to complement televised instruction and to ini- 
tiate appropriate group instruction), and parent instruction (to promote 
positive parent-child interaction, to facilitate the use of home instructional 
materials, and to enable the parent to perform in an effective instructional 
role). Primary and secondary behavioral objectives are explicitly defined in 
the Curriculum Planning Guide . The specific objectives of the television 
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programs, mobile classroom lessons, and home visitor lessons for children relate 
to the categories of 1) orienting and attending skills, 2) motor activity, 
3) language construction, 4) descriptive language, 5) cognition/sensory 
discriminations, 6) cognition/higher order acts, and 7) affect. 

It was originally the intention of the project designers at AEL to develop 
and examine a systein that. If found to be feasible, could then be developed by 
individual school districts or regions or states to meet their own needs. To 
this end, they made available for administrators a set of seven publications 
that describe the elements of the program and the requirements for its imple- 
mentation including costs based on a projected population of 25,000 participants- 
These publications are: Program Overview and Requirements, Field Director's 
Manual , Handbook for Mobile Classroom Teachers and Aides , Home Visitors Hand- 
books , Personnel Training Guide , Curriculum Planning Guide , and Materials 
Preparation Guide . 

During the process of development, however, interest grew in obtaining the 
already prepared basic elements of the program, principally the taped TV broad- 
casts and the materials to be used by home visitors, parents and mobile class - 
rooms. Hence the matter of identifying the costs of this product, or in fact, 
what the actual product is at this point, are clouded by this dual aspect 
regarding dissemination. 

HOPE was field tested for 3 years, from 1968 to 1971, in southern West 
Virginia. Approximately 300 children were involved in the instructional 
groups and another 120 children were assigned to a control group. 

The three main aspects of the program, 1) television broadcasts, 2) home 
visitors, and 3) mobile classrooms, were examined for their effect on the total 
program by assigning children to different treatment groups with some receiving 
only the television instruction, some receiving television instruction plus home 
visitor instruction, and some receiving television, home visitor and mobile 
classroom instruction. According to the outcomes of this study, the mobile 
classroom instruction contributed significantly to gains in social skills 
development, but did not contribute significantly to gains in cognitive develop- 
ment. In general, however, significant gains were found in cognitive develop- 
ment for the experimental groups receiving instruction through the APEP program 
over the control group that did not receive formal instruction. 

In another study in which the performance of children in the HOPE program 
was compared with the performance of children. in a standard kindergarten program, 
the HOPE groups scored higher on the average on the cognitive measures than did 
the kinder gar tv:in groups. On this basis, it was concluded by the researchers that, 
if the. HOPE objectives are appropriate for 5-year-old children, then the HOPE is 
a more cost-effective means of attaining them than a standard kindergarten 
program. The total cost of the APEP was estimated to be about half that of the 
standard kindergarten program. 

There are some b^r^sic administrative concerns in considering the use of this 
product. First, because of a high initial cost for the production of video tapes 
and materials, the program is not economically feasible for use with small groups 
unless these tapes and materials were to be made available as part of the package 
(this, of course, would require careful examination by the user to be certain the 
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objectives underlying development of the materials were consistent with 
identified needs). Secondly, consideration must be given to basic assumptions 
underlying the use of the program. Three of these are that 1) there is 
adequate TV coverage available, 2) there are appropriate locations for mobile 
classrooms, and 3) the potential for parent involvement is high. 

Following is a brief summary of the personnel, materials, and facilities 
required for implementing the program including some basic costs. 

A Field Team , consisting of mobile classroom teachers and aides, para- 
professional home visitOiTo, the field director, and his staff is needed. 
This group initiates and operates the local program. The local program may 
be a single coraraunity, one school district, or several school districts work- 
ing cooperatively. The field office staff includes a field director, an 
assistant field director (depending on the size of the program) , and a sec- 
retary. Specialized knowledge in early childhood education is required. This 
field office staff can manage a maximum of 16 units (150 children each), 
although a smaller number of units is preferred. Each unit of 150 children 
requires one mobile teacher, one aide> and four home visitors, 

A centrally located field office is needed. Basic requirements include: 
office space for the staff; storage room for books and teaching materials; 
heating, cooling, lighting, and ventilation; parking space for mobile class- 
rooms (one per unit of 150 children) ; and space for small group testing 
(desirable but not necessary) , 

Field operation costs are estimated at between $242,15 and $250.33 per 
year per child. (Increasing or decreasing the number of enrollees makes no 
appreciable difference in cost.) 

Field costs for capital outlay for materials and video tapes are estimated 
at $106. ?8 per child (or an annual cost of $21. 68 per child if amortized over 
a 5-year period). The total annual cost of both field operation and capital 
outlay is approximately $263.83 per child (based on 1972 cost figures), 

A Materials Production Center is needed to produce television lessons and 
related materials for all children and parents in the program. These include: 
170 new TV lessons per year; 34 weekly editions each of Parents' Guide , Home 
Visitor Activities , Mobile Classroom Instructional Guide , and Master Curriculum 
Planning Guide ; numerous activities items for children; and feedback and 
evaluation instruments. 

Facilities and equipment for the Materials Production Center consists of 
Office and Work Space: including storage, a conference room, and darkroom 
facilities'. Technical Equipment: 2 cameras (minimum), one video recorder, one 
film chain, and electronic editing; Studio: suitable lighting, at least 3 
permanent sets (e.g., living room, kitchen, and exterior), and occasional use 
of temporary sets. It is. suggested that provision for production centers be 
"arranged by lease from either commercial or educational TV stations. 

The minimum Materials Production Staff required to produce the materials 
noted above includes a director, field-coordinator, procedure-director, on- 
camera teacher, two curriculum specialists, artist-photographer, media 
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specialist, production assistant, and two secretaries. A script writer is 
desirable. Also needed is an operating crew consisting of 2 cameramen, one 
audio engineer, and one video engineer. The staff is based on a small 
geographic area and a relatively smaller number of children. 

As stated, a major obstacle of the dissemination of the process developed 
is the cost of TV lesson production, NIE has provided one hundred thousand 
dollars for the production of two pilot tapes of marketable quality by the AEL 
before June 1, 1973. Further funding by NIE will be considered after NIE has 
reviewed the pilot tapes. If the necessary TV lessons are produced and made 
available to those who wish to implement the process, the rapid dissemination 
of the process will follow. 

The Consortium of Appalachian State Departments of Education has been 
organized by the 7 states served by AEL. The initial purpose of this Consortium 
is to foster the implementation of the Home-Oriented Preschool Education program 
in the tonsortium member states* Final organization meetings were planned for 
January 1973 in Washington, D. C. to create the capability within the State 
departments of education to fulfill the purpose of the Consortium. 
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Elementary School Evaluation Kit:: Needs Assessment 
Ralph Hoepfner, Principal Investigator 
Center for the Study of Evaluation 



Definitely 
Recommended 
(Dollar Support)^ 



B' 



Conditionally 
Recommended 
(Dollar Support)'* 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



Action Prior 
to Disseminaiion 
(Dollar Support)^ 



Definitely Not 
Recommended 
(No nr>llar Support)'* 



PANEL DISCUSSION 

A comparative evaluation should be made with such products as PROP-AC-144 (Book No, 28) - 
Determining Instructional Purposes. The product should be combined with PROP-AC-173 
(Book No. 14) - Evaluation Workshop I: An Orientation - and dissemination recommendations 
made in terms of the common portions first and the unique parts separately. The product 
will be a good model for a number of people. NIE should insist on its production at 
lower cost Although no funds should be allocated for cost reduction); the existence of 
any residual government copyright should be investigated in this connection. A low-cost 
version should come up for PROP Preview in 1974. 



I'ANE-LIST COMMENTS 

Does not need dissemination money. 

Good to have a systematic approach to evaluation. 

Too broadly conceived — dangerous. 

Omission of coverage on certain areas may have had ill effect. 



NOTATION 



1 • Condition to be satisHed is indicated in Panel Discussion, 

2 - Resubmission in 1974 is suggested. 

3 - K = Independent Evaluation or Independent Review of Available Data, T = l ield Trials, M = More Development, 0 = Other Action as 

Specified. 

4 • **DolIar Support** rel'ers to Dissemination Dcillars, 
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Elementary School Evaluation KIT: Needs Assessment 



Principal Investigator: Ralph Hoepfner 



Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 



The Elementary School Evaluation KIT; Needs Assessment is intended to 
provide a tool for the elementary school principal who wishes to identify 
those educational goals that are considered important by himself, his staff, 
and the community and that need to receive greater emphasis in his school* 
The KIT includes, for each participant in the assessment, a set of 106 cards, 
each identifying a possible educational goal, generally expressed in terms 
of one or several related behavioral objectives; four blank cards for additional 
goals; five mats onto which cards are sorted, labelled on a continuum from 
"unimportant" to "most important"; and a tally sheet to record how each goal 
was rated, with the goals listed in alphabetical order. In addition, the 
principal receives tally sheets to average the ranking of goals and forms on 
which to enter other necessary computations. A questionnaire including 
the same goals is available to be used in place of the card-sorting method. 
The manual offers detailed plans for sampling teachers, parents, and community, 
including sample letters and memos designed to win cooperation- After the 
top-priority goals are identified by the method described, the school popu- 
lation is assessed in these skills. Available norm-referenced tests for each 
of the goals at each grade level are listed and rated in the manual according 
to their measurement validity, examinee appropriateness, administrative 
usability, and normed technical excellence. Tables for finding the school 
norm as well as a differentiated school norm that adjusts a school's score 
according to its socioeconomic, racial, and geographical characteristics are 
also included. At this point, mathematical calculations are employed to 
determine which goal should be chosen for improvement, taking into account 
its priority ranking, the value and probability of improving student performance 
in that goal area (information provided) , the current level of student 
performance, and the typical level of student performance. The entire process 
is very carefully justified and explained. CSE feels that the KIT provides 
enough information to let the principal do a job he would once have had to 
hire a consultant to do for him. The case studies suggest that it should take 
one school year to administer and interpret the results of the Needs 
Assessment. 

The stated objectives of the field test were: " 1) to determine whether 
various procedures contained in the KIT had been implemented by the principal; 
2) to determine any changes that had occurred in the following areas that 
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could have resulted from use of the KIT — a) the attitude of the principal 
and his staff toward evaluation, b) the methods used to make decisions relative 
to the instructional program of the school, and c) the understanding of the 
principal and his staff of the evaluation principles on which the KIT is based; 
and 3) to determine the subjective opinion of the principal and his staff 
toward the contents of the KIT." (From the CSE Report No. 70, p. 10.) In- 
formation obtained was used to improve the KIT. 

Seventy-nine schools nationwide, as well as 103 schools in California, 
participated in the field testing. An effort was made to achieve a represent- 
ative sample in both the schools nationwide and^ the schools in California. The 
schools nationwide had requested participation in the project; the California 
schools paid $1000 each to the Association of California School Administrators, 
which was running a project on accountability of which this was a part, to 
participate; thus one may assume that all schools were positive about their 
involvement. The school principal was required to complete 5 questionnaires 
related to school characteristics and to the 5 parts of the KIT. Results are 
reported separately for the national sample and the California sample. In 
addition, 2 case studies were completed in participating California schools. 
These studies included the administration of a sociometric device to identify 
influential teachers, in order to determine the effect of their attitude 
toward the KIT on the staff in general, and interviews with the principal and 
some participating teachers and parents. The results of the field testing 
led to some modifications of the KIT. 

The KIT, costing $89.95, includes: a manual, tally and computation sheets, 
a set of sorting mats and decks of cards in a number equal to the number of 
people to participate in the largest group to be involved in card sorting, and 
enough questionnaires for all the parents to be polled. There is enough 
material for an average-sized elementary school. Additional material is 
available. 

The KIT was scheduled for publication by Allyn and Bacon (Boston) on 
January 1, 1973. 
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CSE-ECRC Preschool /Kindergarten Test Evaluations 
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Definitely 
Recommended 
(Dollar Support)'^ 



Conditionally 
Recommended 
(Dollar Support)"* 



1 L J I 1. 



Promising But 
Not Yet Ready 
(No Dollar Support)^ 



IMNII 

Action Prior 
to Dissemination 
(Dollar Support)^ 



i I I i t t 



Definitely Not 
Recommended 
(No Dollar Support)'* 



PANEL DISCUSSJON 

Users of this product should understand that the additive nature of the scale used in 
evaluation may hide serious deficiencies in individual tests and that the validities 
of the tests were determined by fairly arbitrary methods. The product's main value is as 
a catalog, since the evaluations presented are limited. Training is required in the use 
of the product. 



PANELIST COMMKNTS 

Needed at this time. Kany teachers of preschool classes are looking for instruments. 
A systematic approach, but the evaluations are subjective. 
The field test was not appropriate. 
Undesirable side effects are suspected. 

Ideally, [what is needed is] a book giving real good validated appraisals of 
[each] test's usefulness and validity, including, ideally, how much its use improves 
students' educational attainment. 

How does the book compare with Buros's Mental Measurements Yearbooks? 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = I 'ieJd Trials, M = More Development, O = Other Acrion us 

Specified, 

4 • '*DoUar Support" refers lo Dissemination Dollars. 
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CSE-ECRC Preschool/Kindergarten Test Evaluations 



Principal Investigator: Ralph Hoepfner 

Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 



CSE-ECRC Preschool/Kindergarten Test Evaluations is-a critical and 
objective guide to all published assessment, diagnostic, and prognostic 
instruments for preschool and kindergarten children and is intended for 
use by principals, counselors, and directors to select tests that are 
valid, appropriate, usable, and technically sound. The final compilation 
covers approximately 120 tests, including over 630 subtests with separately 
normed scores • 

The 79 objectives in the taxonomy of goals of preschool and kindergarten 
education were obtained through interviews of practitioners and early child- 
hood specialists as well as an exhaustive search of programs and the litera- 
ture. These represent a comprehensive statement of preschool and kinder- 
garten education. Tests were then keyed to these educational objectives and 
evaluated by measurement experts and educators on the 24 criteria of the 
MEAN evaluation system. 

MEAN is an acronym for a system develviped by CSE which reflects 4 critical 
areas of concern to test users: 1) measurement validity: content and con- 
struct, concurrent and predictive; 2) examinee appropriateness (comprehension, 
content, and instructions); format (visual principles, quality of illustra- 
tions, time and pacing, and recording responses); 3) administrative usability: 
administration (test administration, training of administrators, and aC linis- 
tration time); interpretation (norm range, score interpretation, norm groups, 
score conversion, score interpretation, and the possibility of making decisions); 
4) normed technical excellence (stability , internal consistency, alternate 
forms, replicability , range of coverage, and gradation of scores). 

Each test is categorized as a preschool (30 to 59 months) and/or a kinder- 
garten (60 to 72 months) measure according to the publishers' claims and 
given a numerical rating for each of the criteria and a summary letter grade 
(£ood. Fair, or P^oor) for each of the 4 main categories. The subcharacter- 
istic ratings are considered to be additive and as such may obscure drawbacks 
in the instruments. The developer points out that few instruments aimed at 
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this level can be considered to be adequate or appropriate for this population. 
The initial portion of the book describes the objectives and point-grade 
procedures for each criterion. In the evaluation section, the instruments are 
identified by title, section, and publisher and located in alphabetical order 
under the appropriate goals which are ordered alphabetically by primary gool. 
An index of goals and tests and a list of addresses of publishers are included. 

No field testing of the materials is planned. There will be an examination 
of the impact which the product has on people who have used the evaluation, in-- 
eluding a sampling of the people who have bought the book. This report will 
parallel the report on the field testing of the CSE Elementary School Evaluation 
Kit (CSE Report No. 70). Although not yet published, it is now in final form 
and should be printed by February 1973. 

Personnel who use this book should have experience or training in school 
administration or school evaluation. No other expertise is required of the 
people using the book. 

The book costs $5.00 and is available from The Center for the Study of 
Evaluation of Instructional Programs » It is to be revised and updated every 2 
to 3 years. A supplement will also be published containing the prices of all 
instruments. Dr. Seligman of the Center indicates that there are no plans to 
have this product published commercially. Evidently there had been some initial 
contacts made, but publishers were not interested. The Center has also felt 
that it is performing a service that could be done better by a nonprofit 
educational organization. 

The Center has distributed about 3,000-4,000 copies to school administrators, 
child development centers, and organizations concerned with early childhood 
education. Materials from the Center have been reviewed in several educational 
journals and this has aided in the dissemination of the prodi-ct. 
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Definitely 
Recommended 
(Dollar Support)** 
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Conditionally 
Recommended 
(Dollar Support)'* 



_l L t 1 111 1 i 



Promising Bui 
Not Yet Ready 
(No Dollar Support)** 



iimx! 

Action Prior 
10 Dissemination 
(Dollar Support)^ 
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Definitely Not 
Kecom nu'nded 
(No Dollar Support)** 



PANEL DISCUSSION 

Users of this product shouJd understand that: the a-ditlve nature of the scale used in 
evaluation may hide serious deficiencies in individual tesr.s and that the validites of 
the tests were determined by fairly arbitrary methods. The product ^s main value is as 
a catalog, since the evaluations presented are limited. It is preferable that the 
product be used by those with training in evaluation. 



PANELIST COMMENTS 

The sytematic appro^c^h to evaluating tests in the school setting is much needed. 
These evaluations are subjective. 

How does this book compare with Euros 's Mental Measurements Yearbooks? 
Field-test data are lacking. 

Ideally, [what is needed is] a book giving real good validated appraisals of [each] 
test's usefulness and validity, including, ideally, how much its use improves students' 
educational attainment. 

No dollars. 



NOTATION 



1 • Condition to be satisfied is indicated in Panel Discussion. 
2- Resubmission in 1974 is suggested. 

3 • E = Independent Evaluation or Independent Review of Available Data, T = Field Trials, M = More Development, O = Other Action as 

Specified. 

4 • "Dollar Support" refers to Dissemination Dollars. 
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CSE Elementary School Test Evaluations (ESTE) 



Principal Investigator: Ralph Hoepfner 



Center for the Study of Evaluation 
UCLA Grruuate School of Education 
Los Angeles, California 90024 



ESTE is a comprehensive appraisal of 1649 published tests for elementary 
school students. Each test is keyed to one of 145 educational objectives 
and is rated on each of the 24 criteria of the MEAN evaluation system, the 
TP^in catep;ories of which are measurement validity, examinee appropriateness, 
auministrative usability, and nomed technical excellence. The ratings are 
presented for each of grades 1, 3, 5, and 6 in a grid of educational 
objectives versus rating criteria. Each instrument is identified by title, 
section, and publisher and is located in alphabetical order under the objec- 
tive to which it is keyed. Following the numerical ratings, each test is 
given a summary grade: ^ood. Fair, or Poor, for each of the four main criteria 
In the introduction, the exhaustive list of educational objectives which CSE 
developed are outlined and the MEAN criterir and point-grade procedure are 
described in detnll. ESTE was designed to provide information for making 
decisions regarding assessment devices and to be used primarily by school 
administrators and secondarily by counselors, researchers, state and federal 
agencies, and test producers. 

CSE views the advantages of ESTE as "conciseness, currency, educational 
relevance, objectivity, and consistency" (p. vi) . This product is an expanded 
version of one of the 5 instructional chapters which comprise the Elementary 
School Evaluation KIT: Needs Assessment, first stage of the CSE School 
Evaluation Project model. The 5 chapters of the first stage are Description, 
Choosing Goals, Selecting Tests, Collecting Information, and Selecting Critical 
Needs Areas. Similar products are planned for 4 other evaluation stages: 
Program Planning, Implementation Evaluation, Progress Evaluation, and Outcome 
Evaluation. Additional versions of the evaluations have been developed for 
preschool and secondary education.' 

Several studies relating to the evaluation of the product have been 
published. These are: 



1. Ralph Hoepfner, et al. , "Report on the. Field Testing of the CSE 
Elementary School Evaluation KIT: Needs Assessment/' CSE Report 
No. 70, September 1971. 
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Pertinent to ES^Ewfere the responses of principals to questionnaire #3 on 
Booklet III: Selecting Tests. (ESTE is an expanded version of Booklet III of 
the KIT,) Over 80% of the principals in both samples rated Booklet III good or 
very good on clarity, organization, appropriwACc:ness , usefulness, and length, 

2. F, K, Hassenstanun, '*A Survey of the Users of CSE Elementary School 
Test Evaluations," CSE Working Paper No. 19, August 1971. 

Questionnaires were sent to everyone who purchased the book and, of the 
2413 sent, 697 were returned in time to be used in the study. The responses 
were fairly consistent across the categories of respondents and in SMinmary the 
response was considered positive. 

3. Ralph Hoepfner, ''A Test of Tests,'' CSE Report No, 69, May 1971. 

In this report, the data obtained from a factor analysis performed on the 
24 ratings given each test under the MEAN evaluation system are tabulated and 
analyzed. It was determined that the 4 consistent dimensions upon which tests 
vary are usability, norm quality, focus, and psychometric quality. 

4. R. Hoepfner and W. Doherty , "What Test Publishers Publish," CSE Report 
No. 75, February 1972. 

Using the ESTE data, analyses of variance and cluster analyses were per- 
formed; the results were significant* The obtained publisher profiles and 
clusters are presented in the report and are intended tc be of assistance to 
both test purchasers and test producers. 

Several questions have been raised about the ratings in ESTE. Dr. Richard 
Seligman, the assistant directpr of the project, said that the listed authors 
were the raters and described the rating procedures as follows: Each test was 
independently rated by a minimum of 3 raters and if they did not agree, other 
raters were consulted. Each rating is not an average, but a debated agreement. 
However, since only one of the 4 authors was a psychometrician, all of the 
raters for any given test could have been graduate students. It would seem 
important to note which of the ratings were made by whom. 

The ratings on the "Measurement Validity" criterion appear to be almost 
entirely subjective. The authors themselves say, "Empirical measurements of 
such validities were not demanded for the evaluations, although they were 
desirable." (p. x) However, according to the APA Standards (1966), "To examine 
construct validity requires a combination of logical and empirical attack," 
(p. 13) The first part of the validity rating termed "Content and Construct" 
validity is so subjective that the allowance of as much as 10 points for it is 
questionable. The second part of the validity rating, termed "Predictive or 
Concurrent" validity, is also subjective, but to a lesser degree. For this 
rating it is noted that manuals were consulted but that "No attempt was made to 
comb the research literature for additional or more supportive findings." 
(p. xvii) Fully 15 points out of the tonal of 60 (or one-fourth) are allocated 
according to these primarily subjective procedures. 

There is some question with regard to the assertion that these ratings can 
be used by persons untrained in psychometrics • There are, of course, problems 
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inherent in additive rating systems in general. The assumption is made that 
ratings on criteria are additive. But what if, for example, the internal 
consistency reliability on an instrument were zero? In the ratings, it could 
still turn cut relatively well provided the ratings on other criteria were 
good. The point is that certain aspects of instruments may be crucial, and 
an additive rating model makes no allowance for such. 

To compare ESTE ratings with reviews in Buros's Mental Measurements 

Yearbooks , one of the highest and one of the lowest rated tests from ESTE 

were selected. It was found that ESTE's evaluations were not always consistent 
with Buros's evaluations. 

There are no special personnel requirements or other administrative con- 
siderations implicit in the use of ESTE. The product is a paperback book which 
can be obtained from the: 

Center for the Study of Evaluation 
UCLA Graduate School of Education 
Los Angeles, California 90024 

The cost of the book is $5.00 and no other costs are involved in obtaining 
or using the product. 

ESTE was released in 1970 by CSE prior to the completion of the field 
testing of the KIT. To date approximately 7,000 copies of the book have been 
sold by direct mailing. Dr. Seligman reports that although the test evaluations 
were not originally planned for separate dissemination, he believes that ESTE 
should continue because its audience is broader than that of the KIT, which 
focuses entirely on the principal. In the future, brochures will be sent to 
a broad population, and complimentary copies will be sent to chief state 
officers. 

The entire Elementary School Evaluation KIT: Needs Assessment was 
published by Allyn and Bacon in January 1973. 
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Perceptual Skills Curriculum 

Robert Glaser and Jerome Rosner, Principal Investigators^ 
Learning Reserach and Development Center (LRDC) 
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De finitely 
Recommended 
(Dollar Support)^ 
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Conditionally 
Recommended 
(Dollar Support)"* 



Promising But 
Not Yet Ready 
(No Dollar Supporl)"* 



Action Prior 
to Dissemination 
(Dollar Support)'* 
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Definiiely Not 
Recommt'nded 
(No nollar Support)^ 



PANEL DISCUSSION 

The early data appear favorable. It would seem that aiding development might be justified, 
since this would broaden the base for field trials beyond the middle class and a single 
inner-city area. This product could become an interesting prospect and allocation of 
funds for the purposes specified might speed its development to the point where it was 
ready for dissemination. 



PANELIST COMMENTS 




NOTATION 

Only 7 of 10 panelists present during final balloting. 

i - Condition to be satisfied is indicated in Panel Discussion. 
2- Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = I'ield Trials, M = More Development, O = Other Action as 

Specified, 

4 - "Dollar Support'* refers to Dissemination Dollars. 
^ Educational Testing Service, Princeton, New Jersey, February 1973 
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Perceptual Skills Curriculum 

Principal Investigators: Robert Glaser and Jerome Rosner 

Learning Research and Development Center 
University of Pittsburgh 
Pittsburgh, Pennsylvania 15213 

The Perceptual Skills Curriculum is a developmental program whose major 
aim is to prevent learning disabilities from arising in children simply because 
the perceptual skills of those children have not been developed adequately. The 
curriculum assumes that certain perceptual skills are necessary to academic 
success; that they can be taught; that they can be arranged hierarchically and 
taught systematically in units with clearly defined behavioral objectives: and 
that, once taught, they will contribute to the academic growth of the child. 

The curriculum covers four general areas: Visual-motor activities, 
Auditory-motor activities. General-motor activi.ties, and Letters and Numerals 
(emphasizing the academic application of the other skills) . Using criterion- 
referenced diagnostic tests, generally individually administered , the teacher 

first establishes the level of skill that each child has attained in each area. 
Then, to develop the child's skills from those levels on, the teacher leads each 
child through the activities appropriate to his level of development as they 
are outlined in the various activities manuals. 

The curriculum consists of: a) Curriculum tests, b) A Visual-motor activi- 
ties manual, c) A-ii Auditory-motor activities manual, d) A General-motor activi- 
ties manual, e) Vijual-motor manipulative devices (including a Design Board, 
on. which the child copies specified designs by placing rubber bands on pegs, 
and Printed Patterns, on which the child draws the required designs), f) Auditory 
motor instructional cassette tapes, g) Letters and Numerals instructional 
cassette tapes, and h) Record-Keeping Forms. 

The curriculum has been tested mainly on children in kindergarten and fi-^st 
grade; however, it is claimed that the General-motor tests and the lower level 
of the Visual-motor and Letters and Numerals tests can be used effectively with 
3-year-olds, although the Auditory-motor tests are not recommended for 
children younger than 4 1/2. The materials would therefore also seem appropriate 
for use with individual children or e^ren small groups of children receiving 
instruction outside the school. »In addition, che materials . will have some 
value in remedial work to correct the kinds jf problems that the developmental 
program is attempting to prevent. ' 

A series of studies was undertaken during the development and validation 
of the Perceptual Skills Curriculum. First, those perceptual skills that appear 
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to be directly related to the basic classroom tasks of reading and arithmetic at 
the primary-grade level had to be identified. Standardized perceptual tests 
(the Gesell Copy Forms, the Rutgers Drawing Test, the Word Repetition Test, and 
the Auditory Organization Test) were administered to approximately 300 kinder- 
garten and first-grade pupils in urban and suburban Pittsburgh at the beginning 
and at the end of the academic year 1969-70; the scores on these tests were 
related to the population's scores on the Wide Range Achievement Test, Since 
these correlations were deemed significant, it was decided that the study should 
not only be replicated but also carried a step further: to identify the aspects 
of the skills most relevant to reading and arithmetic. To measure more refined 
and complex aspects of the skills than were identified in standardized tests, 
special tests were developed in Auditory-motor skills (AAT) and Visual-motor 
skills (VAT). (Work is still under way in General-motor skills.) The tests 
were given in 1970-71 to pupils in the Baldwin-Whitehall Public School District 
(Pennsylvania), A comparison of results on these tests with results on standard- 
ized tests (Language Arts, Word Reading, and Arithmetic subtests of the Stanford 
Achievement Test) showed positive correlations. 

Next, the task* of training pupils in the identified skills began. In one 
of three similar studies that are part of the validation study conducted for 
Visual-motor skills, individuals in Class A (14 preschool children in an urban 
Pittsburgh school that was participating in a Primary Education Project) received 
special training in the use of the Design Board; no one in Class B (15 preschool 
children from the same school) did. At the end of about 47 days (an average of 
about 13 days in pupil-training time), individuals in Class B were trained for 
approximately the same length of time as those in Class A. (Class A received no 
L rther intensive training.) At the beginning there was no statistically 
significant difference between the mean score for Class A on the Rutgers Draw- 
ing Test and that for Class B. Following the training of Class A (but not Class 
B) , the mean score for Class A was higher than that for Class B. (In a possible 
score range of 0-8, Class A had a mean of about 5.5 and Class B of about 3.) 
After Class B was trained, a statistically significant difference between the two 
groups no longer existed. In four months? the Mean Equivalent Drawing Age 
(scores on the standardized Drawing Test) of the two classes had risen from less 
than 48 months to 58 months. 

A study of similar design was done foi; Auditory-motor skills, as part of 
the validation. The results indicated that the experimental group receiving 
special training made much more progress in Auditory-motor skills than the 
untrained control group. 

Another study indicated that a group of children trained in auditory skills 
reached the level of ability (or perhaps surpassed it) of an untrained group of 
children from the same inner-city neighborhood who were approximately a year 
older. 

No study was done for General-motor skills; the developer, however, suggests 
that studies in the fields of physical education and physical therapy indicate 
that such skills are highly trainable. 

No new studies were conducted as part of the project to determine whether 
training in perceptual skills would affect achievement in reading and arithmetic, 
but reference is made to other studies on that point. 
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Finally, the various aspects of the training program, the Perceptual 
Skills Curriculum itself, were validated in terms of their ability to work 
in the classroom — to show changes in the behavior of pupils going through the 
program. The assumption was that the pupils exposed to the training would 
advance along the hierarchy of skills and pass the appropriate criterion- 
referenced tests for the skills if the program were adequate and feasible. 
In Visual-motor skills. Auditory-motor skills, and General-motor skills, the 
mean performance on the criterion-referenced tests of 57 kindergarten children 
in the Baldwin-Whitehall district who began the program in September 1971, by 
January 1972, approximated or surpassed the beginning performance of 32 first- 
grade children who had also been tested that September. 

The data appeared favorable enough to the panel to justify possibly aid- 
ing development in order to broaden the base for field trials beyond the 
middle class and a single inner-city area and thus bring what the panel con- 
sidered an interesting prospect to the point where it was ready for dissemina- 
tion. 

The teacher using the curriculum should be able to include it in the normal 
classroom routine. However, because so much of the instruction is individual, 
the use of paraprofessional help is probably advisable. The teacher should be 
familiar with the various activities and tests before they are used with the 
children and should be ready to prepare materials (such as ditto copies of 
mazes) or collect materials (such as bottle caps, cubes, and blocks) to supple- 
ment the materials supplied in the curriculum. The classroom should be pro- 
vided with one or more cassette tape recorders. 

The developer reports that he has talked to various publishers about publish' 
ing the curriculum. One publisher estimated that to supply the complete set of 
materials in their original format to a school would cost about $700 a set. 
Very shortly, however, the developer expects to sign a contract with the Walker 
Educational Book Company (New York), which plans to publish the materials on 
May 1, 1973 in six volumes. The first volume will include (in permanent bind- 
ing) the Introduction, Rationale, Validity Study, and Guidelines. The other 
volumes will be in ring binders to permit the materials in those volumes to be 
used as master copies that can be duplicated easily by the school. Volume II 
will include the Visual-motor tests and activities, (Two Design Boards in each 
of three configurations will also be supplied, but others can be purchased 
separately.) Volume III will include the Auditory-motor tests and activities; 
scripts for tapes will be included, but not the tapes themselves. Volume IV will 
include General-motor tests and activities. Volumes V and VI will include 
Letters and Numerals tests and activities. Presented in this form, the publisher 
estimates a price of $50 for all 6 volumes and 6 boards. A school need buy only 
one original set. 

In discussing the ways in which funds for dissemination could be most prof- 
itably used, the developer poin^.ed out that the company that intends to pub- 
lish the curriculum is a small one with limited funds to spend on dissemination 
and that the plan for publishing the materials was devised to lower the cost of 
the curriculum and thus bring it within the financial scope of more schools. 
Therefore, funds for dissemination would be most advantageous in that they could 
be used to promote the product, to make the public aware of its existence. 
Furthermore, funds could be used to provide workshops and training in the use of 
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the curriculum. The only other way to provide such workshops > suggests the 
developer, is either to raise the price of the materials to cover the cost of 
the training (something those connected with the project are reluctant to do) 
or to keep the price of the materials low and charge for participation in the 
workshops (adding to the financial burden of the schools). 
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PANEL DISCUSSION 



The product should be put in wide circulation. Let people know about it to give, it 
even greater visibility* Omission of the affective area is a serious lack. 



; 

PANELIST COMMENTS 

By definition, calculated to provide side effects. 
Limited dollars. 

Not only a tool for decision-making and change, also a reference source. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion, 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = Field Trials, M = More Development, O = Other Action as 

Specified. 

4 . '^Dollar Support" refers to Dissemination Dollars. 
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A Sourcebook of Elementary Curricula Programs and Projects, ALERT 



Principal Investigator: Samuel N» Henrie 

Far West Regional Laboratory for Educational 

Research and Development 
1855 Folsom Street 
San Francisco, California 94103 



A Sourcebook of Elementary Curricula Programs and Projects is one of the 
products of ALERT (Alternatives for Learning Through Educational Research and 
Technology), an information system being developed by the Far West Regional 
Laboratory. This 493-page volume describes approximately 300 programs in 
elementary education. Included are the following categories: aesthetics and 
art, affective education/personal development, career education, drug education, 
early childhood education, English/language arts, environmental education and 
ecology, ethnic education, intergroup relations, foreign language and bilingual/ 
bicultural education, health, sex and family life, physical education, mathe- 
matics , reading, science, social studies , general systems , and resources . 
There is a cross-referencing to permit educators to locate either in its table 
of contents or index appropriate programs as several can be listed in more than 
one category, for example, in both ecology and science. Each area is further 
subdivided in order to provide information about four kinds of educational 
development: i) curricula, ii) models, iii) training, and iv) resources. 

The sections for each program include a general description which., while 
indicating its major aspects, also includes not only its strong points but 
also limitations and in some cases, major criticisms. Brief comments are given 
under the following headings: target audience, subject area, content emphasis, 
sample topics, major goals for students, project goals, suggested use, length 
of use, unit sequencing, instructional methods, student's role, teacher's role 
and training, student testing, late-entering students, special equipment and 
facilities, program evaluation, availability, present status, cost of materials, 
equipment, and services; current information about status, distributor, and 
developer. All of these are brief enough for educators to gain a maximum 
amount of information in a minimum amount of time. 

The purpose of the Sourcebook is to provide information to decision-makers 
in schools about new available alternatives in curricula, training programs, 
models for school organization, and instruction methods, giving priority to 
those developed through federal grants and through research and development 
institutions. Programs and projects selected to be included in the Sourcebook 
are limited to 1) those which serve the needs of preschool and elementary 
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level children; 2) those available in some form within one year of release of 
information; 3) products which are "well-developed," that is, have adequate 
funding, competent development groups, and sufficient scope and quality to be 
of use on a national scale. 



Specific priorities for inclusion in the Sourcebook are: 

First Priorities ; programs produced by rigorous research and development methods; 

i.e., establishment of performance objectives, field testing 
of the parts and complete program, revision on the basis of 
test results, and insistence upon adequate performance before 
release for publication. 



Second Priorities ; validated new nonresearch and development programs that have 

not been rigorously evaluated but which show internal evidence 
of good quality and represent important alternatives to tradi- 
tional practice. 

Third Priorities : programs with innovative features or other new programs that 

have not been rigorously evaluated but which show internal 
evidence of good quality and represent important alternatives 
to traditional practice. 



Details of these priorities and a listing of programs with indications of prior- 
ities are given in the Technical Report on the Selection of Kntries for the 
ALERT Information Product, A Sourcebook of Elementary Curricula, Programs, and 
Projects . 

Initially the descriptions of programs were printed on punched cards and 
separate pamphlets, but the first main field tests indicated that a catalog format 
would be more economical and feasible. 



In the initial version, which consisted of punched cards and separate pam- 
phlets, approximately 100 test sites were selected with 3-8 individuals completing 
questionnaires before and after the installation of ALERT at each site. Two 
surveys of site coordinators were carried out during the year. The purposes of 
the field tests were to 1) determine whether the product could meet its perfor- 
mance objectives and 2) learn how ALERT is used by school personnel; 

These data were used in a formative context in order to provide clues for 
the improvement of the information system and its products. A result of this 
first field test was to change the information given on punch cards and pamphlets 
to the present Sourcebook version in catalog format. A second field test for 
1971-72 was carried out because certain critical questions were not conclusively 
answered within the first year findings (for example, one of the chief objectives 
of the ALERT system has been to facilitate adop tion/rejection decisions and data 
in this regard were "favorable but inconclusive"). Furthermore, this second year 
of testing might reveal the capacity of ALERT to hold the interest of users over 
time and to keep them continuously abreast of new developments. 

V 

The picture emerging from the second field test is that ALERT is a useful 
and desirable information tool, an important part of an information/decision 
process, but not in itself sufficient to facilitate that complete process. It is 
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valuable in that it provides a general reference to what is new in the elementary 
curriculum, a source for initial search, a source of information about specific 
programs, a tool for screening programs with particular characteristics, and a 
teacher-training text. Teachers need direct experience in looking at materials, 
talking to teachers, and seeing the programs in operation. For ALERT to be a 
complete decision-making system would require that sample materials be made 
available and site visitations be possible so that potential adopters could have 
some direct, concrete experience. In the quality control aspect of the field 
testing, minor inaccuracies were discovered, but they were never enough to 
cause a reader to gain a distorted picture of the program or products. 

Major details of the field testing have been published in Report On the 
Second Year of the Main Field Test of the ALERT Information System , which is a 
supplement to the report on the initial field testing. The technical reports 
have been extremely candid in revealing not only the types of field testing, but 
also those aspects of the program which did not initially fulfill expectations. 
The emphasis on formative evaluation and subsequent changes made as a result of 
this are carefully described. 

There are no specific personnel requirements for this product except that 
administrators and teachers concerned with elementary education would be in posi- 
tions where this information might prove extremely profitable in a survey of 
current programs preparatory to making educational decisions. 

The Sourcebook is used to determine what is available in elementary educa- 
tion. If decisions are to be made, materials for specific programs are necessary. 
Possibly a site visit for direct experience with those using the materials and a 
determination of the specific conditions under which the program would be used 
most effectively would be needed. The Sourcebook is available from the U.S. 
Government Printing Office, Washington, D.C. 20402, Stock Number 1780-1072, at 
a co'jt of $5.75. 

The Sourcebook is being sent free of charge to those people who have helped 
in its development, provided information, or encouraged the project. Copies are 
also being made available to selected key educators. Continuation of present 
dissemination efforts with some publication of information as to the availability 
and description of the Sourcebook are planned. 
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PANFL DlSCU^iSION 



This is a well done package in an area that needs help; the cost is low. Training in 
the use of ERIC and other information resources is a need that is well met through a 
product such as this, especially the book component. The audiovisual materials may 
need to be shed, but if they are retained, independent evaluation or field trials 
arc called for. 



PANELIST COMMENTS 

Goals well defined. 

Evaluative data from users not exposed to the course would be useful. Not completely 
for independent use. 

Need validity data showing that, after the course is taken, the consultant gives 
better service as judged by the client. 

Wonder about market — is audience limited? 

How does this product relate to competing products in training in the use of ERIC 
such as the CASEA computer retrieval system? 

For information seekers rather than supervisors. 



NOTATION Q^j^y ^ q£ panelists present during final balloting — an earlier vote was 9 x B 

J 'Condition lo be sansfiec: is indicated in ^-nel Discussion. 1 X C if E is done. 

2 ' Resubmission in 1974 is >;uggesled. 

3 - E = Independent Evaluation or Independent Review of Available Data» T ~ I-ield Trials. M = More Development. O = Other Action as 

Specified. 

4 - **Dollar Support" refers to Dissemination Dollars. 
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1855 Folsom Street 
San Francisco, California 94705 



The primary purpose of this instructional product is to train Educational 
Inforaation Consultants (EIC's) so that they can receive, process, and fulfill 
client requests tor information related to specific curricular, instructional, 
and administrative problems. The ETC provides a link between the client and 
those information networks (for example, ERIC) which now exist to stockpile 
educational research and development information. The need for such a 
course is based on the assumption that the knowledge is more valuable when 
it can be effectively communicated between those r»^sponsible for its 
generation and those responsible for its utilization. 

The primary form of the instructional system is a 30-hour (ten 3-hour 
sessions) training course entitled, "The Education Information Consultant: 
Skills in Disseminating Educational Information." The course is organized 
in 5 modules ranging ii length from 4.5 to 9 instructional hours. Each 
module focuses on one of the 5 major processes of the EIC role: Negotiation, 
Retrieval, Transformation, Communication, and Evaluation. 

Each instructional module begins with a clear statement of objectives. 
For example, those stated for the "Retrieval" module include defining and 
explaining the significance of the retrieval process; developing an efficient 
and comprehensive search plan; and knowing how to conduct searches in a 
variety of educational resource systems, including the ERIC system. 

The instructional activities were originally designed to f:'.t the 15- 
session pattern of a semester schedule. They have been compressed, however, 
into 10 sessions, a pattern which is compatible with a quarter-term schedule. 
The developers plan to make the course available in 3 other delivery forms: 

1) a 45-hour course for university use 

2) a 10-day self-contained institute 

3) a self-administered learning team form for 3 or more trainees 

The initial training materials were developed following a comprehensive 
survey of the literature and a 2-day conference in which working papers and 
models were t^resented to a panel of consultants. Representatives from ERIC 
ana other education il centers attended the conference. Three tests of the 
instructional T^aterials have been made: 
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1) a preliminary field test (PFT) of the instructional system 
at the University of California, Berkeley, 

2) a main field test (MFT) at San Francisco State College, and 

3) a preliminary operational field test (POFT) at the San Mateo 
County Educational Resources Center. 

A total of approximately 40 persons participated in the field tests 
which were conducted in 1S71. They represented experience in a variety of 
occupations: librarian, information analyst, director of library science 
program, assistant to school superintendent, teacher, research chemist, 
and graduate student in education. During the field tests both cognitive 
and affective data were collected. 

The cognitive evaluation was designed to determine the extent to which 
the knowledge and skills required to perform the EIC role were acquired 
following instruction. For the preliminary field test a combination objective- 
subjective test was administered as a diagnostic exercise at the beginning and 
end of the training sessions. When pre- vs. post-comparisons were made using 
2 raters for both pre- and post-scoring, significant results (i.e., greater 
mean scores) were noted. Two modifications of the system following the ^^FT were 
the elimination of certain oral presentations and more flexibility in 
scheduling. 

The MFT involved 18 graduate students in education at San Francisco State 
College. For the MFT, the pre- and post-test consisted of in-ucuse multiple- 
choice, matching, completion, and short-answer subjective items. Again highly 
significant mean score differences indicating improvement in performance were 
found. In addition to the overall evaluation, separate modular tests were 
used on 4 of the 5 modules at each of the first 3 field trials. 

The affective evaluation consisted of administering at the end of each 
LiOdule questionnaires containing affective scales. This procedure was used 
in both the PFT and MFT. The scales and criteria were developed by the project 
staff. For the PFT, the set criterion was achieved on 75% of the items and for 
the MFT the criterion was achieved on 80% of the items. 

There were complaints that the course schedule was too tight. In fact, the 
field tests compressed a course designed for 45 hours into 30 class hours. 
Instructors felt that the student activity built into the course was a positive 
factor for the program. 

The preliminary operational field test (POFT) was conducted at the San 
Mateo County Education Resources Center in Spring 1971 to test the feasibility 
of the system as a transportable package. Nine trainees already functioning 
at various levels in the EIC role were involved. The primary purpose of this 
field test was to determine if the instructor materials and guidelines were 
sufficiently developed for indapendent use. Tests similar to those described above 
were used and the final project report contains several quotes from participants 
attesting to the value of the course. Those trainees with considerable ex- 
perience reported "significant" gains in learning in the Negotiation and 
Communication modules. Few trainees showed a gain in the Retrieval module since 
most were quite familiar with ERIC. 
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A major conclusion following the POFT was that the present form of the 
EIC course is not completely adequate for independent use. It was felt that 
the instructor needs some training prior to teaching the course. 

If the course is offered in a university, an instructor with some background 
in education or educational research would be desirable. No special facilities 
are required to implement the course and many individuals can probably master 
much of it via self study with no formal instruction. The developers claim the 
course can be taught by a teacher using the ^^Instructors ' Manual." This manual 
has detailed lesson-by-lesson guides. 

There are a variety of materials related to the course. A looseleaf 
notebook containing job aids, readings, learning exercises, guidelines, and 
other accessories developed for student use costs $15. Instructor materials 
consist of the student notebook ac' ompanie'u by a schedule and detailed notes 
on the format arid content of each 3-hour instructional session (cost about 
$10). Although not specifically stated, an adequate resource center (library) 
is probably an asset in developing the skills and goals outlined for the 
course. 

Audiovisual materials now built into the instructional time include 2 
filmstrips with tapes, an audio-tape, and a slide-tape set. The slide-tape 
set describes the ERIC DIALOG system. Copies of these materials are 
available in the Far West Laboratory and would be provided in tx.^ transportable 
package (no cost data available). Two films — one, 12 minutes and the other, 
^ 22 minutes in length — are optional. Neither is available from the Far West 
Laboratory but both can be rented, at nominal cost, from film distributors 
who give nationwide service. 

Wliile the EIC process could be applied to many different educational 
audiences, the content and examples in the course seem primarily directed 
to EIC's x^^ho v^ill work with teachers and curriculum supervisors concerned 
with subject-matter information at an elementary and secondary level. The 
degree to which the EIC can serve effectively other segme.its of the 
educational community (e.g., at the postsecondary level and/or in adminis- 
trative-policy level) will depend on the extent to which the skills developed 
through the course examples are generalizable . 

Both the negotiation and communication steps would probably be facilitated 
if the EIC can function in the client's professional language and terminology 

The EIC product focuKes on published materials and literature. This may 
lack the timeliness needed in information for policy-level decisions. There 
could be a unit on extracting and transforming data available only in undigested 
form, e.g., HEGIS data and state-level school census data. The negotiation, 
transformation, and communication skills of the EIC process would be applicable 
to such undigested data but the retrieval pro..ess might be quite different. 

The demand for professional EIC*j may be somewhat lacking and employment 
restricted to places like the ETS ERIC Center, large city school systems, 
educational R and D Centers, State Departments of Education, and the San Mateo 
County Education Resources Center. However, the product should not be 
evaluated solely on the basis of career preparation, but rather viewed as a 
short course that may improve one's ability to serve as a resource person. 
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For example, a university library could have a staff member take the short EIC 
course and thereby improve its service to users. 

The product is presently being disseminated by the Far West Laboratory. 
Beginning next fall (1973), the University of California will offer a correspon- 
dence course based on a modified version of the program. The course was 
demonstrated at the 197?. AERA convention over a 5-day period. Approximately 125 
people (not inc2.uding AERA) have taken the course. According to the Final Report 
a proposal was submitted to the USOE for additional funding for evaluation and 
dissemination. The Final Report is available from ERIC: ED-055-610. 
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PANEl Dl'-CUSSION 

Not only should an absolute evaluation be carried out, but a comparative evaluation w? th 
competing products, notably PROP-AC-96 (Book No. 01) - Elementary School Evaluation Kit: 
Needs Assessment - and PROP-AC-173 (Book No. 14) - Evaluation Workshop I: An Orientation, 
should be made. The product has the notable and rare feature of showing means and 
confidence limits as well as gains and probable errors. It appears somewhat more manageable 
than PROP-AC-96. The evaluation of the product was good, but the criteria used could be 
more rigorously defined via independent comparative evaluation. 



PANELIST COMMENTS 



What is the educational significance of the standards used as performance criteria? 
Skill needs better external evidence of significant gains. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 
2- Resubmission in 1974 is suggested, 

3 - E = Independent Evaluation or Independent Review of Available Data, T = hield Trials, M = More Development, 0 = Other Action as 

Specified. 

4 - "Dollar Support" refers to Dissemination Dollars. 
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Determining Instructional Purposes is a training package containing a 
Coordinator's Handbook and training units on 1) Setting Goals, 2) Analyzing 
Problems, and 3) Deriving Objectives. 

The purpose of the training units is to provide coverage of those skills 
that are basic to instructional planning and management. Each unit represents 
one step in a comprehensive process for translating information about 
instructional problems, goals, and constraints into systematic plans for 
the instructional program and consists of 1) reading materials, 2) simulated 
input, and 3) training exercises. The units use several methods, such as 
teamwork and feedback. The objectives for each unit are as follows: 
1) Goal Setting (10-20 hours, 3 modules) - to guide instructional planners 
in determining what a district's goals are and what they mean, 2) Analyzing 
Problems (17-18 hours, 7 modules) - to enable instructional planners to set 
priorities for action on district problems, and 3) Deriving Curricular 
Objectives (8 hours, 4 modules) - to enable planners to translate their 
goals into appropriate objectives and to develop mechanisms for assessing 
those objectives. 

Although the units fall into logical sequence, they have been designed 
so that they may be taken in any sequence or each one can be used alone. 
When the units become operational, they will not require the presence of 
the developers. Each unit contains all materials and instructions necessary 
for a coordinator to conduct a training course and is appropriate for a 
variety of training schedules and settings. 

The units are intended for 1) school planners at elementary and secondary 
levels, 2) local school administrators, and 3) teachers. The "entry level" 
of trainees need not be high because the skills covered are relevant to 
school people having a wide range of duties. 

The field test for Setting Goals was conducted at 5 sites with 74 
participants between October 1971 and Fc^bruary 1972. Two Preliminary 
Performance Tests and 3 Definitive Performance Tests measuring 6 cognitive 
and 4 skill objectives of the training unit were given. All participants 
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completed an individual written 2-3 hour pretest which was later readministered 
as a post-test. Trainees also completed a questionnaire on Affective Objectives. 
Trainees rated on a 7-point scale the degree to which the skills and knowledge 
gained from the training unit would be useful to their work. Results were as 
follows: 1) All the cognitive objectives specified for the introduction were 
achieved. 2) Cognitive and skill objectives specified for Module Two were 
met. 3) Cognitive objectives and product specifications for Modules One and 
Three were achieved, but the results were not positive for skill objectives. 
The performance effectiveness for these Modules has not been demonstrated. 
4) In terms of affective objectives and product specifications appJied to the 
entire unit, the results were positive. With the exception of skill objectives 
for Modules One and Three, the Goal Setting unit was found to be effective. 
Several alternatives are offered for skill objectives in Modules One and Three. 
The standard of performance x^as typically the midpoint of the possible score 
range for each objective. For an objective to be considered achieved, the 
following conditions were to be met: 1) the pretest group mean was below 
standard; 2) the post-test group mean was above standard; 3) the post-test 
group mean was significantly greater than the pretest group mean. Once release 
is approved, additional performance testing can be arranged. 

Three Preliminary Performance Tests of Problem Analysis were conducted 
between March and July 1971 with 44 trainees at 4 sites. An individual 
written pre- or post-test was administered before and after training. A split 
sample pre-post design was used and the evaluation of results was based on the 
objective that performance on the post-test would be significantly higher than 
on the pretest in terms either of the percentage of correct responses on items 
which could be judged correct or incorrect or the quality of the responses in 
terms of criteria presented in the Guidelines as judged by independently 
trained raters who were unaware of the field test design. After training, 
trainees also completed a knowledge test to evaluate their achievemer?t of 
cognitive objectives and a questionnaire to assess thair affective reactions 
to training. The performance effectiveness of the Introduction, Modules One- 
Four and Six-Seven was demonstrated in the performance field test. Module 
Five did not prove to be effective during performance field testing. Since 
Module Five is not essential to the entire unit, it was eliminated. Generally, 
performance field test results showed that product users acquire basic skills 
and knowledge related to the problem analysis process, show readiness to apply 
the process, and respond favorably to the product. 

The main field test for Deriving Objectives was conducted from October 
1971 to March 1972 at 5 sites. The sample ranged from 12 to 168 participants. 
One version of. the test focused on foreign language curricula and one was a 
standard version. Both were administered to graduate classes in education, a 
professional organization workshop, and to foreign language teachers in a 
school district. A pretest and post-test were used to determine whether 
training objectives were achieved. All participants took the same pre- and 
post-tests. The training objectives were broken down into 25 discrete test 
variables. Achievement of statistical significance was deemphasized in favor 
of monitoring the practical significance of achieving a noticeable pre-post 
test gain on each variable. Data were analyzed in a "goal-free*' manner for 
assessment of the direct effects of the training. Affective data were gathered 
to determine the acceptance value of the unit. Participants in the foreign 
language version gave slightly higher affective ratings to the unit, possibly 
because of a common participant interest. Trainees achieved statistically 
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significant post-test gains on 18 of 25 variables tested. The unit 
was considered successful in developing knowledge, comprehension, and 
low-proficiency skill for performing tasks and functions involved in 
deriving curricular objectives. Analysis of affective data revealed that 
unit characteristics described by 128/ of 142 items had met or surpassed 
the standard of acceptability. 

Further revisions of the unit will^ focus on identifying more specific 
skills to be achieved. It should be noted that the entire training package 
will be revised before it becomes operational on a large scale. 

A coordinator who assumes responsibility for the course is required. 
He needs no special training other than what is outlined in the Coordinator's 
Handbook. A group of people involved in local school planning at the 
elementary and secondary levels should participate in the course. Since 
this is an intensive course, a small group may be preferred. However, the 
group should be large enough to accommodate teamwork situations. 

The cost is minimal. Each unit is self-contained and costs approximately 
$8.00. A district may buy 1, 2, or 3 units in the package and reproduce 
the materials or buy enough units for each participant. It is recommended 
by tue developer that the products not be copyrighted and released through 
a commercial publisher be<"ause !:his procedure would increase the price 
substantially ($25-$30 per unit or $75-$90 for the three). Further, the 
product would not be attractive to a publishing house because of the thin 
potential market and the cost of reaching it. 

No special facilities are required. Any sufficiently large room 
would be appropriate. 

The coordinator and participants must schedule a block of time for 
the course in conjunction with district school boards. 

The package is now used in a few districts and about 40-50 requests 
for the units have been received. The materials are disseminated to users 
below cost in addition to handling expenses. 

Promotional literature was sent to school districts around the country 
in February 1973. It is also recommended by the developer that consideration 
be given to securing funds to cover production and free distribution of about 
3,000 units to selected sites where they will be used and that the units be 
placed in the public domain for distribution at a minimal cost through the 
Superintendent of Documents. 
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DefinitcJy 
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Action Prior 
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^Optional $ for distribution where needed 



Definitely Not 
Keoommended 
(No Dollar SuppDrt)'* 



PANEL DISCUSSION 



This is a useful handbook* it is informative and looks quite good. It could 
be given a boost with modest funding. 



PANELIST COMMENTS 



NOTATION 

3 panelists abstained from vo tings 

1 ■ Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suc^gested. 

3 - E = Independent Evaluation or Independent Review of Available Data» T = Field TriaJs, M = More Development, O = Other Action as 

Specified. 

4 - "Dollar Support" refers to Dissemination Dollars. 
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Early Childhood Information Unit 

Principal Investigator: Stanley Chow 

Far West Laboratory for Educational Research 

and Development 
1855 Folsom Street 
San Francisco, California 94103 

The Early Childhood Information Unit is designed for teachers, adminis- 
trators, parents, and community people who are responsible for reviewing, 
selecting, and implementing early childhood programs. Its purpose is to inform 
these decision-makers about educational programs available for young children 
from the preschool years through third grade. It does not recommend any one 
program because it is felt that decisions regarding the adoption and/or 
adaptation of programs ought to be made by the local school community in the 
light of local constraints, conditions, and resources. It is instead the 
purpose of the unit to provide knowledge ureful to those involved in selecting 
programs for young children. 

The Unit's printed materials include an introductory paper and directions 
on how to use the unit, a paper on trends in early childhood education, detailed 
reports of 8 major early childhood programs, less detailed summaries of 7 ot.her 
programs, and a bibliography. Other materials include a set of 9 filmstrips 
and 4 accompanying cassette tapes. The filmstrips include one intended primarily 
for parents which explains major trends in early childhood education. The 8 
major programs are further described in the other filmstrips. 

The reports for the 8 major programs are each approximately 20-35 pages in 
length and contain detailed information on the following topics: goals and 
objectives, content and materials, classroom activities, parent involvement, 
professional and paraprof essional training, administrative requirements and costs, 
program development and evaluation, and the history of the program. 

The other 7 program summaries are approximately 3-6 pages in length and 
cover the same general topics as those listed for the major programs. These 7 
programs were not considered suitable for more detailed treatment because some 
are still in the early stages of development, one is a model for parent-controlled 
schools rather than a classroom program, and another is a model for training 
parents , 

The programs given detailed treatment include Behavior Analysis Model, 
Bank Street Model, Cognitive Curriculum, Demonstration and Research Center for 
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Early Education (DARCEE) , Education Development Center: Open Education Model, 
Responsive Model, Tuscon Early Education Model (TEEM), and the Engelmann-Becker 
Model. 

The programs about which information is presented in summary form are: 
Independent Learning Model, Florida Parent Education, East Harlem Block Schools, 
Primary Education Project, Early Childhood Learning System, Appalachia Preschool 
Education Program, and Ameliorative Program, 

Unit developers hope that users will be able to evaluate rationally informa- 
tion about alternative programs and that they will pay attention to information 
about the programs, respond to it, and value the unit. Specifically, users of the 
Unit will know the major characteristics of programs they review, will understand 
information about them, will be able to identify those that meet their needs, and 
will be able to make judgments about them. 

The evaluation of the Unit included two major field-test stages: a prelimi- 
nary field test (PFT) which was carried out as a formative evaluation and as a 
test of product feasibility and a main field test (MFT) which was a test of the 
Unit's effectiveness in meeting stated objectives. 

For the PFT, a sample of 15 potential users participated in a full-day 
review of a preliminary form of the Unit which included 3 program reports and 2 
abstracts. The subjects considered the Unit to be as valuable as primary or 
direct sources of information, e.g., site visits, pilot projects, and more 
valuable than secondary or indirect sources. They felt the Unit was valuable as 
an information resource, a training ,tool, and a decision-making aid. 

The Main Field Test (MFT) took part in 2 phases, one in September 1970 and 
one in September 1971. 

The MFT Phase I sample consisted of 23 administrators and other profes- 
sionals, 13 parents, and 30 teachers and student teachers. There were 31 sub- 
jects in California, 13 in Nevada, and 22 in Utah. The three groups filled in 
6 questionnaires covering each of the Unit components and another questionnaire 
that asked their opinions of the materials. ^ 

Minimum and optimum standards were set for each objective for each \group 
for each Unit component. In general^ the data show that for the^ knowledge 
objective the standards were easily met. Parents did not perform as well as 
teachers or administrators. The developers found this reasonable, ?ince they 
assume that pvof essionals would be more knowledgeable about the field. With 
regard to comprehension, again parents did not do as well as the other two groups, 
while for the value objective, it was found that the Unit generally was con- 
sidered valuable and that administrators tended to give the highest ratings. 

Phase II of the MFT had as its sample 31 subjects who reviewed the Unit in 
3 separate groups. Group I consisted of 15 San Francisco Bay Area parents, 
teachers, and administrators who are involved in e/=»rly childhood education. \ 
Group II contained 9 people, teachers, parents, principals, and a curriculum 
coordinator, all from a single suburban school district in the Bay Area. This 
second group wa^s considered a homogeneous group because it was^ felt that they 
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could represent a "real" conunittee organized to review early childhood 
programs and also because they were from the same district. Group III con- 
sisted of 1 administrator, 3 teachers, and 3 parents selected from the 
Laboratory's contact file. It was unlikely that they would be drawn together 
as a curriculum committee so they were known as the heterogeneous group. 

Group I reviewed audiovisual elements. They were pretested on their 
familiarity with programs included in the Unit and then proceeded to review 
the Introductory Slidetape and 5 program briefings. Questionnaires and 
knowledge tests were administered as post-tests. Results of cross tabulations 
indicate that the briefings helped the subjects to learn the characteristics of 
those programs with which they had been unfamiliar. 

Groups II and III were pretested and were then instructed to review the 
Introductory Slidetape and all program briefings. After their review, they 
were instructed to select 1 or 2 programs that would meet their needs. Group 
II (called homogeneous) easily reached a unanimous decision. Eight of 9 sub- 
jects in Group II indicated that the Unit provided enough information to make 
a good decision. 

Group III did not reach a decision, but the developers felt that the fact 
that the group was heterogeneous and had no common mission could help account 
for this failure- 
Each program briefing was rated by the respondents on a 7-point scale, 
with 7 indicating high quality, in terms of several characteristics, e.g., 
interest, usefulness, sufficiency of information, ease of use, and clarity. 
The results of the evaluation indicate that :wo programs, Tuscon and Cognitive 
Curriculum, were considered good in all categories, while for the Introductory 
Briefing, DARCEE, and EDC , a number of value categories were marked "fail." 
Revisions were made according to these reactions. 

In Group Il's reactious to the complete Unit, all characteristics except 
for sufficiency of information were considered better than adequate, while 
Group III felt- the Unit was interesting and useful but did not rate it high 
in other categories - 

An Operational Field Test was conducted in December 1971, using a sample 
of 22 persons, 3 administrators, 6 curriculum specialists, 5 teachers, and 
8 parents, in four westerri states who were asked to review all materials. 
They were then organized into review committees charged with recommending a 
course of action. Questionnaires were filled out by all participants. 

Data obtained include; demographic data, results on tests of knowledge 
and information objectives, results on tests of application and decision objec- 
tives, results on tests of value objectives, and results on tests of cost- 
benefit-analysis objectives. A typical exercise, such as one applicable to the 
application and decision objective, asked that subjects first list 5 important 
problem areas encountered when considering programs for implementation and then 
indicate 1 or 2 Unit programs that might solve the problem. Thirteen (59%) 
of the 22 subjects indicated that the Unit's programs would solve at least 3 
of the 5 problem areas l-^'st^id. Five (23%) did not respond. From these results, 
the developers felt that the subjects were able to apply Unit information. 
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On the basis of the various tryouts, the program developers felt that their 
unit helped fulfill a need for making concerned persons aware of what is avail- 
able in early childhood programs. 

The Unit can be used by either a single person or a group of persons, 
although it is intended for use with groups of administrators, teachers, parents, 
or any combinations of these. No special training is required for the group 
leader but he should become familiar with the materials before presenting the 
Unit to others. 

The printed materials available consist of 5 booklets. Introduction and 
Directions, Bibliography, Review of Trends in Early Childhood Education, and the 
Detailed Summaries of the 7 other programs. Other materials include 9 filmstrips, 
i.e.. An Introductory Filmstrip intended primarily for parents and 8 other film- 
strips that deal with the 8 major programs covered in the Unit. Each was photo- 
graphed in a classroom using the model recommended by the developer. 

All materials are required for full understanding of the Unit which can 
be purchased for approximately $90.00, although costs and prices are not con- 
sidered fixedo The Unit may also be available in the future on a rental or 
lease basis. This purchase is a one-time cost and, since a trained and experi- 
enced educator can learn to present it, no major changes in staff will be 
aecessary . 

Presenting the Unit requires a fil^nstrip projector and a cassette tape 
player. Most school districts are likely to have these available; otherwise 
rental costs will be dependent on the locale. 

It is suggested that presentation of this Unit be done in at least two ses- 
sions. If a presentation is to be made to a parent group only one session may 
be needed if the leader feels that all 8 filmstrips need not be shown. Presenta- 
tions may be made at school staff meetings, district staff meetings, PTA meetings, 
or inservice meetings. 

The Early Childhood Information Unit will soon be in the hands of Educational 
Products Information Exchange (EPIE) , a nonprofit educational concern, which will 
be the sole distributor. No specific information about thiir dissemination 
strategy is available at this time. 

Far West Laboratory personnel will still be available as information sources 
and in consultative capacities. 
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Defi nicely Not 
Recommended 
(No Dollar Support)*^ 



PANEL DISCUSSION 



This is a useful kind of product and will not cost much to disseminate. It 
describes kinds of programs that should be 1 rought to the attention of educators, 



PANELIST COMMENTS 

Very urgent present need, at least in principle. 
Notably original scanning guides and layout. 

Information on this v-o^^ct should be disseminated, but direct funding should 
be limited- 



NOTATION 

Only 9 cf 10 panelists present during final balloting. 

1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = Field Trials, M = More Development O = Other Action as 

Specified. 

4 - "Dollar Support" refers to Dissemination Dollars. 
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The American Government Information Unit 



Principal Investigator: C. L. Hutchins ^ 

Far West Laboratory f/^o: Educational Research 

and Development 
1855 Folsom Street 
San Francisco, California 94103 



The American Government Information Unit is a paperbound volume of 
information about 9 recently developed secondary school American Government 
programs compiled to f.^cilitate comparisons of the features of the various 
products. The prograns Included nitt: 7 criteria: the materials were appropriate 
for American Governmer.t classes; they could be used for a semester or more of 
study; they were desi/ned for students in grades 9-12; student materials had 
been developed; teachers* guides were also available; the materials had been 
field tested in high school classes and revised on the basis of test results; 
and the materials were available for purchase in 1171. All of them involve 
some degree of "inquiry** and ^'discovery" and the skills and processes used 
by social scientists. The 9 products selected and reviewed are: 



1. Utah State University Social Studies Project ( A Curriculum Focused on 

Thinking Reflectively About Public Issues ) 

2. High School Curriculum Center in Government Project ( American Political 

Behavior course) 

3. bociological Resources for the Social Studies ( Episodes in Social Inquiry 

Series ) 

4. The Harvard Social Studies Project (AEP Public Issues Series ) 

5. Holt Social Studies Curriculum ( Comparative Political Systems course) 

6 . Lincoln Filene Center Secondary Social Studies Program 

7. Law in American Society Foundation ( Justice in Urban America se'^'xes) 

8. Educational Research Council Social Science Program (ERCSSP) : Concepts 

and Inquiry ( The Price of Freedom course) 

9. The Amherst Project. 



A review of the programs begins by using a screening aid which enables the 
user Lo select sets of desirable features and to eliminate the programs which do 
not have ul^ose features. Examples are: length of use, cost per student per 
year, and stucent ability level. The number uf programs to be reviewed in 
depth can be reouced still further by using a one-page foldout matrix in which 
various dirnensions of the products are compared; e.g., instructional strategy, 
sample topics, and teacher-training requirements. The second section contains 
brief, c^e-oage descriptions of the p'-.oducts* The t.h±Ld section contains 
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detailed, approximately 20-page analyses on the following aspects of each product 
1. Goals and Objectives, 2. Contents and Materials, 3. Classroom Strategies, 
4. Student and Teacher Prerequisites, 5. Implementation: Requirements and 
Costs, 6. Prograni Development and Evaluation, and 7. Project History. 

A preliminary field test in February 1969 was carried out in the Laboratory 
using 19 San Francisco Bay area secondary school teachers and administrators. 
After further development, the main field test was conducted using a sample of 
48 respondents from 9 school dit:tricts. The results showed that the Information 
Unit was successful in achieving its objectives and respondents were able to make 
acceptable decisions, but there were some criticisms of format. The developers 
then decided to combine a retesting of the objectives with the final operational 
testing of the product. Thid was done in the spring of 1970 (Combined Main/ 
Operational Field Test). Two groups were used — one group reviewed the unit with 
laboratory supervision and direction; the other used it independently. 

The following results were obtained: 

a) Average ratings by subjects placed the product at 5.0 or higher 
(toward the positive end) of 7-point scales of "useful" and 
"easy to use." 

b) Given a list of possible resources for curriculum information, 
subjects indicated that they preferred the product to all other 
comparable secondary sources (hiring consultants, professional 
meetings or conventions, and journals) and almost equivalent to 
primary sources (workshops using new American Government curriculum 
materials, site visits to innovative projects, and conversations 
with involved professionals whose judgments they valued). 

c) Subjects perceived that the reward for using the product in terms 
of usefulness, personal benefit, and enjoyment (an average of 
5.61 out of 7 over three reward scales) justified the time and 
cost involved (an average of 3.62 out of 7 over three cost scales). 

More detailed information about the evaluation carried out by the developer 
may be found in the "Final Report' on the American Government Information Unit," 
by N. C. Adelson, S. G. Crosby, and L- A. Sikorski, Far West Laboratory for 
Educational Research and Development, Berkeley, California, January 1971. No 
independent evaluation has been done. 

One copy of the American Government Information Unit is needed per school. 
The cost is $7.95 per copy, available f;:om Technicon Educational Systems. 
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Definitely 
Recommended 
(Dollar Support)"* 



^ondiiionally 
commende-i 
(Dollar Suppo t)4 



Promising But 
Not Yet Ready 
(No Dollar Support)"* 
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(No Dollar Suppi>r()'* 



i>ANEL DISCUSSION 



If the preliminary findings of large gains stand up, the product vyovld appear to be 
very promising » Field testing or an independent evaluation should be encouraged to 
augment the rather skimpy data. Tests of significance of the reported gains are needed. 



PANELIST COMMENT 

Extend the group slowly and double- check results. Not yet sure enough for wholesale 
dissemination. 

Good side effects — excellent results. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = Kield Trials, M = More Development, O = Other Action as 

Specified. 

4 - **Dollar Support" refers to Dissemination Dollars. 
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An Intensive Training Curriculum for the Education of 
Young Educable Mentally Retarded Children 



Principal Investigator: 



Sheila Ross 

Palo Alto Hedical Rese.arch Foundation 

860 Bryant Street 

Palo Alto, California 94301 



This product is "an intensive training curricuxum for the education of 
young menfa] ly retarded children." The curriculum includes: basic academic 
skills (reading, vocabulary, verbal expression, arithmetic); general learning 
skills (listening and following dd.rections, planning, problem solving); social 
behavior (appropriate verbal and nonverbal behavior in commonly occurring 
situations involving interaction with others) ; gross and fine motor skills 
(physical education, fine motor skills); fine arts (painting, music); and 
independence training (seac^ork, homework) . 

Th : curriculum is based on mediation theory and social learning theory and 
adapts the research findings and principles from these theories to the special 
needs and learning difficulties of EMR children. The central goal is the 
transmission of mediational skills to enable the children to learn to generate, 
code, and store verbal equivalents of environmental events and use these 
symbolic representations to direct subs^iquenf. behavior effectively. Extensive 
use is made of modeling procedures, the basic assumption being taat intentional 
training (direct instruction by the teachei) combined with the systematic use 
of modeling procedures results in more effective performance than intentional 
training alone. 

The curriculum consists, of three levels: one preschool year, a first 
pri'jiary year, and a second primary year. Each level has completely developed 
detailed lessons for alli subject areas plus the nonperishable objects needed 
(excluding readily available supplies such as clay) . A class using the 
curriculum would have 1/3 of the children working with the teacher, 1/3 with 
an assistant, and 1/3 by themselves during a given 20-*minute period. Relative 
homo.geneity within groups permits each group to proceed through the lessons 
at its own rate; flexibility of grouping permits a child in one group to move 
to another group if his progress warrants it. 

Other features include the use of sixth graders (high status models) as 
aides in various activities, and an emphasis on positive individual reinforce- 
ment (tangible and symbolic rewards) . In addition, positive reward value is 
attached to the class by providing some of the things (e.g., looseleaf notebooks, 
homework, and track meets) whose lack sets the EMR child apart from his siblings 
and peers in regular classes. 
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During the development phase of the first primary year of this product, 
material prepared for the 11 subject areas was tried out to test the efficacy 
of content and procedures for each subject .rea. Next, lessons were developed 
for a 10-month school year. Then the first primary year of the curriculum 
was tried out in two primary E^^R classes (15 studeurs in each) for one year, 
with two additional EMR classes (15 students in each) using traditional 
curricula and forming the control group. The two groups were comparable in 
termb of race, sex, age, IQ, and achievement test scores; both groups con- 
tained a few children ^'ith problems in speech, vision, and behavior, (The 
control group had one child with a motor control probleip and the experimental 
group had one child with a hearing problem,) 

Pretests and post-tests of achievement and intelligence \<rere administered. 
On the Metropolitan Achievement Test the experimental group had a mean gain of 
20,33 points, as compared with a mean gain of 6,50 points for the control 
group; on the Stanford-Bine t Intelligence Test the experimental group had a 
mean gain of 11,63 points, as compared with a mean gain of 0,40 point for the 
control group, (The tests were administered by experienced testers, graduate 
students in psychology, who were told that the purpose of the testing was 
to collect normative data,) 

Class placement figures for the following year also reflect the greater 
success of the experimental curriculum: in the experimental group, 7 children 
went to normal classes full time and 3 part time in September, 1 was scheduled 
to enter normal classes part time in February, and 1 was in a transitional 
class that would enable him to move into an educationally handicapped (EH) 
program the following year, with 18 remaining in the EMR class; in the control 
group, 2 children were in an EH program and 1 was in a transitional class 
that would enable him to move into an EH program the following year, thus 27 
remained in the EMR class. 

It is claimed from the evaluation phase that the theoretical base, 
content emphasis, and procedures were highly effective. Using the identical 
approach to that of the first primary year, a preschool year and a second primary 
year have now been developed. The result is a 3-level program with great 
ccntinu'.ty , with neither overlapping nor omissions in content, requiring no 
unusual or expensive equipment; and in a forra that any reasonably capable adult 
can follow without difficulty. 

One EMR teacher with a degree in special .education and one assistant (this 
assistant may or may pot have experience in instructing children if she is 
interested and able to follow instructions she should be able to follow t'lie 
lessons) are needed in the classroom to work with the curriculum. The material 
provided includes the manual (it is hoped that this will be inexpensive) and 
all of the necessary non-perishable material. Books recommended for use are 
usually already available in the school library; otherwise, they must be 
purchased. 

Pending approval of the copyright by the U. S. Office of Education, the 
curriculum will be published by Lear, Siegler-Searon , 6 Davis Drive, Belmont, 
California 94002. They are a publishing company that specializes in material 
for the handicapped and haxT-e about 30 salesmen on the road, mainly in the 
East and Midwest. 
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Recommended Recommended Not Yet Ready to l)issemlnati<jn Keco Tiniended 

(Dollar Support)"* (Dollar Support)^ (No Dollar Support)"* (Dollar Suppori)^ (Nu DoHar Support)'^ 



PANKL DISC'tJSSION 

Disfjemination should not be contingent upon further evaluation, although the need for 
independent evaluation should be stretised. The lev^l at which the material is presented 
should be checked for suitability to the target population. Although field testing thus 
far involved a small n, significant improvement occurred, at least in terms of the judgment 
of the learners that they had* indeed learned and retained the learning over tirae. The 
product will ^;ork with administrators, but perhaps not with evaluators. 



PANELIST COMMENTS 

Improvements in performance of those tasks for which trained seem substantial although 
not earthshaking. 

Goals narrow but useful. 

Requires some advance planning. 

Would not subsidize anythin& but an inexpensive accurate mailing piece. 



NOTATION 



r 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T = Kield Trials, M = More Development, O = Other Action as 

Specified. 

4 - **Dollar Support** refers to Dissemination Dollars. 
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Project Management II: Basic Principles and 
Techniques of Project Management 

Principal Investigator: C Peter Cummings 

Research for Better Schools, Inc. 
Suite 17U0, 1700 Market Street 
Philadelphia, Peunsylvania 19103 

Basic Principles and Tr ,^iiqups of Project Management is Module 2 of 
the Educational Project Management Instructional System (EPMIS) . It is 537 
pages long, takes about 24 hours (including the supplementary case simulation; 
to complete, and uses 580 slides and 1/ tapes. It can be used by eithei an 
individual or a group. The case simulation which supplements the instruc^iion 
is 1S7 pages long and can be used only by a group. The intended a^^dience is 
current and potential project managers, '^his module is designed to teach a 
project manager to plan, organize, direct, .^d control a project and to attain 
the project objectives within the given ti:^.e, cost, and performance con- 
straints* The developers feel uhat the uie of Modules 1 and 2 is sufficient 
to train school personnel in mc^aaging individual projects. It is also 
possible to use Module 2 without training the top-level administrators with 
Module 1. The chapter headings in Module 2 are: an introduction to project 
management, project definitiou, work flow, time estimation, resource allo- 
cation, cost estimates and budgets, project start-up, developing the project 
information system, problem identification, problem-solving through management's 
actions, solution or decision implementation, and project termination. Target 
audiences other than project managers are identified as: project staff 
personnel, school administrators, school district central office staff, under- 
graduate and graduate educators, community agencies' personnel interacting 
with schools, and supportive staff and agencies having liaison with local 
schools (such as state departments of education). 

The EPMIS instructional materials were developed by Research for Better 
Schools, the Educational Program Management Center (EPMC) of Ohio State 
University, and an advisory committee. The developers first did a task analysis 
of project management and then stated the behavioral objectives of the program. 
In 1970-71 Modules 1 and 2 were pilot tested with 27 school administrators, 
37 project managers and support personnel, and 6 university professors ' 
representing 17 different school districts from 10 different states. The 
objectives of the pilot test were to answer the questions: 1) "Have product 
specifications been met?" 2) "What improvements in the prototype product 
development would increase the level of attainment of product objectives?" 

3) "VJhat are the content, methodological, and media errors and inadequacies 
of the instructional materials?" The product was evaluated and revised. 
Additional testing in 1971-72 used only 6 school districts in 3 states, but 
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involved 50 school administrators, 100 project managers and support personnel, 
and 25 university professors. The 2 modules were again evaluated and revised. 
Modules 1 and 2 are now ready to be disseminated. The developers hope to 
obtain further feedback from questionnaires that will be included with the 
instructional materials. 

The material is self-instructional. However, RBS feels that some school 
districts will need help and is planning to train personnel who can provide 
training and to establish demons tration/ training centers. The manual costs 
a^bout $15; there is no need for one manual per person, Th3 manual for the 
simulation exercise costs about $5; one manual per person is needed. Cassette 
tapes and slides cost aboun $200; a tape recorder, slide projector, and screen 
are required, 

A publisher consultant firm is looking for a publisher for EPHIS, Meanwhile, 
RBS has listed the prodi.ct in some educational publications and is hoping to 
disseminate it Iccally, No obstacles to implementation are foreseen. Additional 
].unds would help the dissemination effort and the effort to obtain further 
evaluative feedback from such dissemination. 
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Definitely 
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Definitely Not 
Recommended 
(No Dollar Support)** 



PANKL OlSt'USSlON 

Edit or remove the entries on those research or try-out tests that have no supporting 
data or are presently unavailable and/or insert a caveat stressing the experimental nature 
of certain of the tests which remain integrated as a nonremovable part of the product and 
noting that they should be dealt with with caution. The question of including or ex- 
cluding each of the instruments should be studied carefully. The product serves an area 
of need wheia not much is available. 



PANELIST COMMENTS 

Good to highlight areas of inadequate instrumentation. 

Book needs editorial revxsalon in light of recognized shortcomings such as those 
listed in caveats 



NOTATION 



1 - Condilion to be satisfied i> indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 • E = Independeiit Evaluation or Indepcr\dent Review of Available Data, T = Field Trials, M = More Development, O = C'.Iier Action as 

Specified, 

4 • **DoIJar Support*' refers to Dissemination Dollars. 
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CSE/HLP Test Evaluations: Tests for High^j-^^der 
Cognitive, Affective, and Interpersonal Skills 



Jt-'fincipal Investigator: Ralph Hoepine?* 

Center for the Study of Evaluation 
UCLA GraJ-uite School of Education 
Los Angeles, California ?0024 



The CSE/HLP Test Evaluations is a single volume composed of evaluation 
ratings of some 2610 scales and subscales considered to test skills in the 
fiigher-order cognitive, affective ^ and interpersonal areas. The evaluations 
were originally completed for use by the Humanizing Learning Program of Research 
for Better Schools, Inc., according to a set of taxonomies developed for that 
program, but they have been published to be 1) of assistance to educators in 
the development and assessment of innovative goals and programs and 2) of 
assistance to researchers, psycl^ometricians , and test publishers in identifying 
areas for which adequate instrumentation does not exisc. 

Each instrument is r^ategorized into a cell of 1 of the 3 taxonomies and 
is given a numerical rating and a letter grade (£ood, I^air, P^oor) on ec.ch of 
the following characteristics: validity, examinee appropriateness (appropriate- 
ness for the intended examinee group), normed excellence, teaching feedback, 
usability, and retest potential, and comments are made on the instrument's 
ethical propriety. The subcharacteris tic ratings are considered to be additive 
and as such may obscure drawbacks in the instruments. The CSE acronym for this 
system is VENTURE .and the first portion of the book is devoted to a discussion 
of the components cf each rating. For example, the system generally gives 
highest usability ratings to instr-iiments wnich may be large-group administered 
in less than 20 minutes by a single observer without extensive practice. It 
will give low normed excellence ratings to a subscale if normative data are 
available only for the whole battery of subscales. 

The instruments are arranged by skill area and subskill, alphabet i^ally 
within subskill. Each skill area is introduced by a description of the 
taxonomy against which the instruments were matched. The cognitive taxonomy 
is a grid of skill (e.g., classifying) against content (e.g., verba] -semantic) . 
That of the affective domain is 3-dimensioaal : affective characteristic 
against type of measure (e.g., self-report) against ty^e of report (e.g., 
speculative). The interpersonal taxonom}- Is 4-dimensional with type of 
relation (e.g.; with peers) mapped against location (e.g., at home) agaiasL 
type of measure against type of report. The instruments are identified by 
name, form, age range, and the initials of the publisher (or author, in the 
case of unpublished tests). An index of tests and an index of publishers are 
included. 
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CSE claims the test e^^aluations offer 1) conciseness, 2) the most 
up-to-date and complete collection, 3) educational icilevance, 4) oojectivity, 
and 5) consistency. 

'^he t/axonomies of the CSE/HLP Test Evaluations V7ere generated in the 
follruxiig manner: the principal investigator and his assistants collected all 
the attributes from existing factor analytic studies (e.g., Braner, Catt-^ll, 
Juilford, and Fiaget) and the developers of the HLP eliminated all factors of 
little interest to their program. An extensive search was then made for both 
published and experimental instruments. Before being rated, each 'nstrument 
was evaluated fo: suitability for use in the HLP. 

Eigh*: raters were employed, ra. ging in experience from graduate assistants 
to a Ph.D. in sevcr.-.O.. fields o^ specialty (e.g., psychological measurement, 
experimental psychology, mt "^surement and testine, educational research, and 
design). All raters ^vere initially acquainted ri'ch the VENTURL system and did 
preliminary ratings on a sample of tests to resolve differences in inter- 
pretation. Each instxuMent wau rated once. However, the ,nost important complex 
and subjective judgmenthi and those judgments of v;hich the .^ater was uncertain 
were checked by at leasJ* one other rater. Ra..ers frequently checked procedure 
and rationale and each checked consistency in his own ratings, Independentxy , 
it has been found that these ratings may be inconsistent with descriptions in 
Buros's Mental Measurements Yearbooks , 

Individual test evaluations are the focus of the product, CSE, however, 
will hc.-^e av/ailable shortly an analysis of the aggregate of tests on each of 
the V/.NlURE evaluative criteria which should give fuller information on cell 
deficiencies to psychometr icians and test publishers. 

No study of the resource value to educators has been made but impact 
studies on 2 similar CSE products (Preschool/Kindergarten Test Evaluations and 
Elementary School Test Evaluations) indicate that the general idea and format 
are useful, 

CSE/HLP Test Evaluations costs $8,50 and is available from: 
Dissemination Office : RBSTE 

Center for the Study of Evaluation of Instructional Programs 

University of California 

405 Hilgard Avenue 

Los Angeles, California 90024 

Brochures describing the evaluations^fwill be sent out to school 
superintendents and, if possible, evaluation officers. Complimentary copies 
will be sent to chief state school officers. If funds are available, a series 
of 1/2- or full-page advertisements may be run in appropriate journals. In 
nationwide travelSj CSE personnel speak about the product. 
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Promising 
Not Yel Ready 
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Dc-finilc.y Noi 
i<t*ciin» mended 
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PANFt DISCUSSION 



The product should be combined with ■PROP-AC-96 (Book Nc - 01) - Elementary School Evaluation 
Kit: Needs Assessment - and dissemination recommendations made in terms of the common 
portions first and the unique parts separately. NIE should insist on its production at 
lower cost (although.no funds should be allocated for cost reduction); the existence of 
any residual government copyright should be investigated.( in this connection. Despite the 
high cost of the total evaluation model, it is good. Since it is essentially simple 
material, it would seem desirable to produce it as a simple low-cost package. The gains 
shown seem weak. 



PANKLIST COMMKNTS 

Deserves exposure despite limited size of taig^^t.. 
Major aid to organizational change. 

No indication that the workshop staff had any clear notion of what evaluation should do. 

Information on this product should be included in any kind of NIE exhibit but no money 
should be given directly. 

I don^t see that the workshop is a **product.'* 



NOTATION 



1 - Condition to be satisfied is indicated m Panel Discussion. 

2 - Resubmission in 1974 is suggested. 

3 - E = Independent Evaluation or Independent Review of Available Data, T - Field Trii.ls, M = More Development, O = Other Action as 

Specified. 

4 r '*Dollai" Support" refers to Dissemination Dollars. " . 
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Evaluatiot: Workshop I: An Orientation 



Principal Investigator: 



Stephen P. Klein 



Center for the Study of 
UCLA Graduate School of 
Los Angeles , California 



Evaluation 
Education 
90024 



Evaluation Workshop I is desi^ined to orient school and state department of 
education personnel to the basic principles, procedures, and problems associated 
with evaluating educational pro^'.rams and to the kinds of information an evalu- 
ation can provide for educational decision-making. The workshop is based on 
the general evaluation model developed at the Center for the Study of Evaluation 
of Instructional Programs at UCLA. It takes 2 days to run and is ideally suited 
for groups of 30 to 45 participants. The workshop materials consist of a leader' 
manual, a notebook for each participant, and a set of exercises for each team 
of 3 participants. 

Each team plays the role of the evaluator in the simulated evaluation of a 
tenth-grade biology-ecology course. The basic instructional procedure for each 
of the workshop's modules involves receiving instruction in one of the 5 phases 
of the Center's evaluation model (such as needs assessmeuL) , practice in solving 
relevant problems in tn±s area via a team exercise (such as determining the 
relative priorities aruong potential program objectives) , and feedback and 
discussion of the correct answers. Instruction is provided via pamphlets, 
lectures, and audio tapes of conversations. By the end of the workshop, partici- 
pants have completed^exercises involving the selection, collection, analysis, 
and reporting of evaluation information for decision-making. 

The team's exercises and feedback materials cr^ packaged in a rip-off pad 
so that each team has a copy of the exercises and each participant has a copy 
of the instructions and feedback materials. Each participant is also given a 
3-ring looseleaf notebook containing all the instructional and simulation, 
material. At the conclusion of the workshop, the exercise and feedback material 
can be added to the notebook to create a 100-page guide to general evaluation 
principles and procedures; the participant keeps this as a reference tool. 

Thr :e versions of the workshop have been field tested; Version 1 at 3 
sites between March 1969 and April 1970, Version) 2 at 5 sites during the spring 
and summer of 1970, and Version 3 at 12 sites between October 1970 and August,- 
1971. The field tests indicate that participants significantly increased their 
knowledge of evaluation, developed a more favorable attitude toward evaluation, 
and found the workshop materials useful in their jobs. Details are repcgrted ^ 
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in: Klein, S. P., and .Jadeau, M. A., "Developing and Field Testing of Evaluatio^i 
Workshop' I: An orientation , " CSE Report No. 71, October 1971. 

Three foritis of a 23-iteni test were constructed and were adminis'ce^ed in a 
counterbalanced design as pretests and post-tests. These tests were Ui^ed to 
evaluate Versions 2 and 3; mean change scores were highly significant in both 
cases. -'or Version 2, the mean pretext score was 14.70 and the mean posc-tect 
score was 16.80, yielding a "t'* valve nf 4.85, which is significant beyond the 
0.01 level. For Version 3, the mean pretest score was 14.69 and the mean post- 
test score v,as 17.15, with a "t" v..iue of 12.67, significant beyond the 0.001 
level. 

At the conclusion of each workshop, a questionnaire was administered to the 
participants so that they could evaluate the workshop and make suggestions for 
its improvement. Both versions were considered quite valuable by most partici- 
pants. For example, 88.^ of the participants indicated that they developed 
solutions to their evaluation problems at the workshop. 

To study the i^pak^t of the workshop on participants who aad taken 
Version 2 or 3, a questionnaire was sent to 297 previous participants in May 
1971. The results were generally favorable. For example, of the 253 respon- 
dens, 72% had used the notebook since the workshop; 67% had discussed or shared 
materials and/or ideas with colleagues; and 90% indicated that the workshop 
enhanced their understanding of evaluation problems. 

The materials required for the Workshop are: 1) che Leader's Manual 
which costs $40, 2) participants' notebooks at $18 each, and 3) a reel-to- 
reel tape recorder. , 

The Workshop requires about 16 hours from the leader and 16 hourf from 
each participant. Participant'^ work in teams of 3. The ideal number of 
participants is 30, but groups o^ 24 to 60 are feasible. 

CTB/McGraw Hill, Del Monte Research Park, Monterey, California began 
publishing and disseminating the Evaluation Workshop in August 1971 ♦ 

! 
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Dorothea M, Ross , Principal Investigator 
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FANKL DISC USSION 



The data show good gains with the use of the product. 



FANtLISr C'OMMKNTS 



Target children began playing with others after 5 months of the program, 
May be disruptive in certain settings. 



DOTATION ^ panelist abstained from voting, 



1 - C'ondilion lo be salisfied is indicated in Panel Discussion. 

2 - Resubmission in 1974 is suggested 

3 - H = Independent Evaluation or Indep<'ndent Review of Available Data, T = l icld Trials, M = More Uevelopment. 0 = Other Action as 

Specified. 

4 - *'DcUar Support'* refers ,o Dissemination Dcilars. 
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Pacemaker Gaines Program 



Principal Inves tiga^tpr : Dorothea M. Ross 

312 Concord Drive 
Menlo Park, California 94025 

The Pacemaker Games Program is a pre-arithmetic readiness program which 
. includes table games and active racing games. It is designed to be used with 
the educable mentally retarded child (EMR) of primary-grade age (approximately 
5-10 years). Through a series of 65 games of gradually increasing difficulty, 
the EMR child learns to participate in informal games designed so that the 
manipulation of numbers is an intrinsic part of the activity. He is intended 
to benefit educationally in a two-fold manner: by acquiring certain number 
knowledge and by learning how to play games. The program differs from other 
similar programs in its emphasis. The child is taught general game skills, 
not numbers. Number knowledge is gained incidentally by playing games for a 
sustained period of 25 minutes for young children and 35 minutes for older 
children. 

Th'^ developer claims the following minimum results (based on research) 
can be expected from use of the program: 1. Mastery of the ten general game 
skills (good behavior during game explanation, playing in turr, taking one turn, 
good behavior between turns, accepting decisions, following rules, being a 
good loser, accepting the end of the game, handling game materials only as 
required, and treating game materials with reasonable care); 2, Ab : ■ ity to 
count hy rote to 10 or higher; 3. Ability to enumerate eight or more objects; 
4, Ability to use ordinal numberals to five or more; 5, Ability to pair the 
members of one set of objects with those of another set; 6, Ability to recog- 
nize small groups to four or more; 7, Ability to recognize common shapes; 
8. Ability to identify colors; 9. Ability to identify coins; 10. Ability to 
use quantitative vocabulary; and 11, Ability to recognize numbers. 

The games are intended to be played with an adult or older competent child 
in charge. This person is referred to as the game controller and his task is 
to supervise the game. A game controller should always be present although the 
c:raount of supervision required should drop sharply as the program progresses • 
The game controller routinely uses number terms such as first, second, next, 
and last, and he thus serves as a model for the children. 

Another adult or competent older child is occasionally brought into the 
game sessions as an adult model. This person's function is to demonstrate 
general game skills that the players are having difficulty acquiring. For 
example, if the players consistently exhibit bad losing behavior, an adult 
model would be brought into one game session, would exhibit bad losing, and be 
O , reprimanded and required to behave properly when losing. 
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The program thus requires one adult (game ccntroller) to supervise the game 
sessions and a second adult (adult model) only when the players are having 
trouble with a game skill. In the 9-month experimental test of the program some 
groups never required an adult model and no group required one more than three 
times. 

Three or four mny play a game nl' one time, but only three if the. adult 
modol is utilized. One game set conlains enough materials for a class of 
30 to participate in different games simultaneously, provided the appropriate 
amount of adult supervision is available; i.e., one adult for every 4 to 5 
children. 

The program was introduced in 1969 and about 500 game sets have been sold 
per year. A total of 1624 game sets have been purchased. Although originally 
developed for the EMR child, the program is now in use in a number' of normal 
kindergarten and primary slow-learner classes. 

The Games Program was validated in a 9-month game training program using 
an Experimental group and a Control group. The subjects were 40 EMR children 
who were free of gross motor^ sensory, and emotional defects . IQ's on the 
Stanford-Binet Intelligence Test' ranged from 51 to 79. On the >asis of 
Chronological Age, Mental Age, and IQ, the children were matched as closely as 
possible in pairs. One member of each pair was randomly assigned to the Ex- 
perimental group and the other to the Control group • The Experimental group 
spent 100 minutes per vieek (3 approximately 35-minute sessions) in a 9-month 
game program, and the Control group spent an equal amount of time in traditional 
special-class number study covering the: same topics. 

At the end of the 9-month training period, the two groups were tested with 
a Number Knowledge test adapted from tests of other researchers. A test-retest 
reliability was established by administering the Number Knowledge test twice to 
a group of 10 subjects in the study, selected from the total group to represent 
the total group. There was a 2-week difference in test dates. The test re- 
liability was 0.98. General game skills were assessed by trained observers. 
Each subject was assessed for a total of twelve 5-minute periods of play with 
three different games and in three different threesomes without an adult model. 
Observer reliability waj computed by the percentage agreement method to be 91 
per cent. 

The Experimental group scored higher than the Control group on the Number 
Knowledge test. Subscores showed that the Experimental group obtained higher 
scores on all measures (rote and rational counting, specific quantitative terms, 
time, money, shape, and color). Minimum results obtained by 18 to 20 members of 
tl-e Experimental group were ability to a) count by rote to 10, b) enumerate 
8 objects, c) use ordinal numbers, d) recognize small groups to four or mora, 
e) pair the members of one set. with those of another, f) recognize common 
shapes, g) identify all U. S. coins, h) identify colors, and i) use quantitative 
vocabulary. The Experimental group also scored significantly higher than the 
Control group on a measure of Spontaneous Quantitative Vocabulary, a subscore 
given by the Number Knowledge test. 

In general game skills, the Experimental group made fewer errors on post- 
experimental measures than did the Control group. This was defined as greater 
improvement in general game skills for the Experimental group. 
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Further support for the Games program has come from parents and teachers 
ui the EMll children completing the program. Letters mention marked increases 
in the use of quantitative terms in both the classroom and free play. Still 
further evidence of success is that the time for administration of the 
Number Knowledge test decreased from three or more sessions in the pretesting 
to one post-test session for lA L'xperimenLal subjects while only 4 Control 
subjects were able to complete the test in one post-tesL session. By the end of 
the fifth month of the program, 14 subjects in the Experimental group were 
playing games routinely with their neighborhood peers or siblings. 

Parents or teachers may administer the program as long as the manual 
is followed closely. Gaines are intended to have an adult (or older competent 
child) controller. In an EME special class, the whole class could play 
the games simultaneously if the .teacher brought in one or two older normal 
children to serve as game controlJ.ers , The program requires one ^ame 
controller for every four or five children. Sixth graders 3 junior and 
senior high school students, and a variety of untrained adult volunteers 
have administered the program successfully. 

Basic material is the Pacemaker Games Program which contains all materials 
for board games. Total cost is $36.00* Action games require some materials, 
usually readily available; e.g., U. S. coins and dollar bills, file cards, 
rulers, cardboard. The materials in the Games Program have been tested for 
durability and are claimed by the developer to be indestructible. Materials 
are readily available in unlimited supply from: 

Fearon Publishers 
6 Davis Drive 

Belmont, California 94002 

In the event of loss of materials, Fearon Publishers will replace some items 
free and require a minimum charge for others. 

The program calls for a minimum of three 35-minute game sessions per 
week. If this is done in a special education class there is no scheduling 
problem. In a regular class where the EMR child is mixed with normal 
children, special scheduling for the teachers and children would be 
necessary. 

At present Fearon Publishers distributes a brochure advertising the 
Pacemaker Games Program and has. about 30 salesmen on the road, mostly 
in the East and Midwest. The developer has plans to revise the teacher's 
manual to include a sample lesson, complete with what to say to the 
children and how to say it. The sample lesson will be used in promotion 
of the product. The publisher has plans to expand the existing 
dissemination efforts to include the West. 
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PANEL DISCUSSION 



Earth: Man's Home 
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Definitely Not 
»!ecommended 
(No Dollar Support 



*Earth: Man*s Home 
**Other components 



Limited funds should be allocated for field trials. Money should be allocated for 
dissemination, but concurrently the considerable data already available should be re- 
viewed rigorously by an independent reviewer. The concept-based approach to curriculum 
has not had a history of marked success. 



PANELIST COMMENTS 

A worthwhile change in approach to the study of geography. 

This program is to supplement, not replace, yet substantial time, presumably taken from 
the existing schedule, is required. It must replace something ^ 

Sample geographically limited. 

Too costly for an unknown competitive gain. 



NOTATION 



1 - Condition to be satisfied is indicated in Panel Discussion. 

2 - Resubmission in 1 974 is suggested. 

3 - E = independent Evaluation or Independent Review of Available Data. T = Field Trials, M = More Development, O = Other Action as 

Specitied. 

4- '*DoUar Support" refers to Dissemination Dollars, 
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Geography Curriculum Project 



Principal Investigator; Marion J. Rice 

University of Georgia 
Athens, Georgia 30601 

The Geography Curriculum Project offers the teacher of elementarj'' social 
studies structured curriculum materials in geography. These units are typically 
Organized to reflect selected major concepts. Organization of materials and 
teaching strategy involve a two-fold emphasis on concept acquisition and concept 
use, based on the premise that any field of loiowledge, such as geography, consists of a 
system of concepts, or word labels, which are used to exjoress ideas and describe 
relationships and that an understanding of geographj^ begins with an understanding of the 
concept system. The units are intended to supplement an existing social studies 
program, not to replace it. 

Earth: Man' s Home, a beginning geography unit designed to develop the geographic 
concept of ''habitat," is for Grade 1 and can also be used at the kindergarten level. 
The teacher' s manual contains geography background information, a section on unit 
development, daily topic lesson plans mth exercises and a suggested lesson procedure, 
and the unit test, which consists of 30 pictorial, three-option items. There is also a 
pupil workbook. The lessons vary in length from 15 to 25 minutes and involve 30-35 
teacher days. 

Place and Environment is a Grade 1 Geography unit. It includes a pupil text, and the 
pupil workbook, the teacher manual, and pictorial test. The primary objective cf this unit 
is to develop the concepts labeled place and environment. There are 12 lesson plans; the unit 
involves 20 teaching days. A third primary-level unit. Res ource and Production , a primary 
unit in cultural Geography, includes a teacher manual, a pupil text and workbook, and a test. 
This unit is based upon two key geographic concepts, resource and production. The unit can 
be completed in 20-25 days, using 20 lesson plans. 

The three primary-level units are designed mainlj^ to be used flexibly as parts of 
an alreadj^ existing social studies program. Those teachers who wish to, however, 
can use the three units together to form a one-semester, primary -grade course on 
Geography. 
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Comparative Rural Landscapes , a five-week unit, wdth 20 daily leai-ning activities, was 
developed for use in upper elementar}^ grades. Tliis unit is organized differentlj^ than 
other units in the series in that it attempts to apply David P. Ausubel' s model of 
verbal reception learning using inclusive theoretical advance organizers to a unit in 
elementary social studies in order to 1) test the hypothesis of the advance organizer 
as a facilitator of learning and 2) to see if the advance organizer increases the 
cognitive achievement of disadvai^taged elementary students. In this unit the advance 
organizer consists of an inclusive, conceptual i^ural landscape model. In Population Growth 
in the United States and Mexico , the concept of population djmamics is applied in a 
systematic unit comparing the growth of population in the United States and Mexico, 
This unit is designed to provide, students in the middle grades wdth a background in and 
understanding of the components of population change. In addition, the unit is designed 
to improve the student' s skill in using maps, charts, and graphs. 

An intermediate-level unit, The Growth of the Black Population of the United States , 
is in its first edition. Two additional intermediate -level units are in the process of 
development and will be ready uext year, an urbim geography unit in the self-instructional 
mastery mode (five alternate forms of treatment of the material have been developed) . 
and a unit on transportation g/aogvaphy. 

The Project has also developed A Bibliography for Geographic Education , a compilation 
of articles, books, and pamphlets published since 1950. 

The Project is claimed to differ from most primary curriculum developments in 
three ways: 1) in providing information ajid experience identifiable with a specific 
dicipline, geography; 2) in being based on a single concept; and 3) in being global 
in scope, rather tlian limited to the immediate environment. The local environment 
serves as the background against which information from distant places is viewed. 

The Geography Currieular Project has been testing each unit for effectiveness as it is 
developed- 

Earth; Man' s Home w as pilot -tested in Spring 1968 with 83 kindergarten children 
in 6 classes in two Georgia school districts. Pupil and teacher feedback, as well as direct 
observation, showed a high level of motivation and interest, notwithstanding the 
didactic nature of the materials. The revised unit was subsequently field-tested in Georgia 
Idndergarten classes. Evaluation of a conceptual beginning geography imit, Esxih: Man^ s 
Home, involved 14 teachers and 268 pupils in 17 classes in four Georgia counties. Unit 
appropriateness w-as evaluated on the basis of its usability by pupils, as indicated by 
measurable learning gains, and by teachers* The evaluation of the unit was based uprn 

1) the reliability and other characteristics of the unit test; 2) the effect of bias 
associated with pupil sex, race, socioeconomic status, and pretesting on geography 
achievement; 3) the effect of bias associated with teacher geography background, 
prior experience with the unit, and teacher perception of the appropriateness of the unit 
on geography achievement; 4) geography gains as indicated by pretest-post-test differences. 
The evaluation indicated that pupil sex was a nonsignificant factor, pretesting was a 
nonsignificant factor, and race and socioeconomic status were highly sigivificant faptors 
with regard to final and adjusted achievement but nonisignificant factors with regard to learning 
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gains. Whites and nonwhites of low and high socioeconomic status made similax' 
gains. Teacher background in geography was nonsignificant, while experience in 
teaching the unit was a significant factor for all four measures. Pex'ception of the 
overall and grade-level appropriateness of the unit were sigiiificaut factors. The 
analysis of pretest-post -test differences on geography leax'ning gains indicated that 
boys and girls, white and nonwhite, of low and high socioeconomic status, regardless 
of the geography background, experience teaching the unit, or perception of unit 
appropriateness of their teacher, made significant gains. 

Comparative Rural Landscapes was pilot tested with intact classes in a Georgia 
school district, controlling for differences in teachers and content identity between 
experimental and control groups. This study attempted to ascertain the comparative 
effects of the use and nonuse of a conceptual model presented as an inclusive advance 
organizer with supple mentarj^ narrative daily organizers on the geography achievemeirt 
of disadvantaged rural black fifth and sixth grade students in central Georgia^ s lower 
Piedmont. The study tested the proposition that carefully structured reception 
learning using inclusive advance organizers facilitates learning among the disadvantaged. 

An evaluation of the intermediate grade self -instructional unit in population 
geography organized according to a method of presentation described as the Forced 
Inferential Response Mode (FERIVI), as compared with a conventional narrative mode 
supplemented with graphics, has been carried out. The content of the two treatments 
was identical and both were designed for use on a self-instxnxctional basis. The study 
also attempted to evaluate the comparative effectiveness of graphics as the primax'y 
source of population information and as a visual supplement to the narrative text. 
All of the classes in the field trials were from three schools in Clarke County, a 
rapidly growing area that is urban in character. The findings of the study suppoxi:ed 
the hy]Dothesis that there was no significant difference in performance between the 
treatment groups on a researcher-constxiicted criterion post -test or in gains on 
standardized tests of skill in reading maps or graphs. FIRM was concluded to embody 
the specific characteristics of the stimulus-response (S - R) reinforcement model 
and may be considered an alternate application to curriculum of the S - R reinforcement 
model. The majority of students in both treatment groups repoxi:ed that they had 
enjoyed working with the unit materials and that they thought the content area was 
interesting and important. 

In all of the testing associated with the Project, whether at the pilot-, the 
field-, or the final-evaluation stage, the Project has used intact classes in 
Georgia public schools. Thus, in each case, 20-50% of the students have been below 
grade level in x'eading and disadvantaged. 
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There are no special personnel requirements. Curriculum materials are available 
from the Geography Curriculum Project. A Classroom Set consists of one copy of the 
teacher materials; and 40 sets of the pupil materials. 



The materials are designed for class units, although thej^ could be used for selected 
individuals, for a whole department, a school, or a district. There are no special 
implications for scheduling or classroom space. 

A brochure describing the materials has been published by the Project staff and they 
respond to mail inquiries. Sales have thus far been very limited. 



Costs: Earth: Man' s Home 

Place ajiid Environment 
Resource and Production 
Rural Landscapes 

A Bibliography for Geographic Education 



$20.00 
35.00 
35.00 
40.00 



(optional) 



2.00 



Population Growth in the United States and 
Mexico 



(no cost figures available) 
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